tmzf5455/EVCS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

skills n8n: garde 5 actives, désactive 3 (config Qwen3.6 local)

Browse Source
This commit is contained in:
Tallagrand Thierry
2026-07-17 00:47:42 +02:00
parent 93aa43d0d0
commit 7a9e56f17c
40 changed files with 22291 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

794
.opencode/skills-off/n8n-code-python/COMMON_PATTERNS.md Normal file
View File

@@ -0,0 +1,794 @@
# Common Patterns - Python Code Node
Production-tested Python patterns for n8n Code nodes.
---
## ⚠️ Important: JavaScript First
**Use JavaScript for 95% of use cases.**
Python in n8n has **NO external libraries** (no requests, pandas, numpy).
Only use Python when:
- You have complex Python-specific logic
- You need Python's standard library features
- You're more comfortable with Python than JavaScript
For most workflows, **JavaScript is the better choice**.
---
## Pattern Overview
These 10 patterns cover common n8n Code node scenarios using Python:
1. **Multi-Source Data Aggregation** - Combine data from multiple nodes
2. **Regex-Based Filtering** - Filter items using pattern matching
3. **Markdown to Structured Data** - Parse markdown into structured format
4. **JSON Object Comparison** - Compare two JSON objects for changes
5. **CRM Data Transformation** - Transform CRM data to standard format
6. **Release Notes Processing** - Parse and categorize release notes
7. **Array Transformation** - Reshape arrays and extract fields
8. **Dictionary Lookup** - Create and use lookup dictionaries
9. **Top N Filtering** - Get top items by score/value
10. **String Aggregation** - Aggregate strings with formatting
---
## Pattern 1: Multi-Source Data Aggregation
**Use case**: Combine data from multiple sources (APIs, webhooks, databases).
**Scenario**: Aggregate news articles from multiple sources.
### Implementation
```python
from datetime import datetime
all_items = _input.all()
processed_articles = []
for item in all_items:
source_name = item["json"].get("name", "Unknown")
source_data = item["json"]
# Process Hacker News source
if source_name == "Hacker News" and source_data.get("hits"):
for hit in source_data["hits"]:
processed_articles.append({
"title": hit.get("title", "No title"),
"url": hit.get("url", ""),
"summary": hit.get("story_text") or "No summary",
"source": "Hacker News",
"score": hit.get("points", 0),
"fetched_at": datetime.now().isoformat()
})
# Process Reddit source
elif source_name == "Reddit" and source_data.get("data"):
for post in source_data["data"].get("children", []):
post_data = post.get("data", {})
processed_articles.append({
"title": post_data.get("title", "No title"),
"url": post_data.get("url", ""),
"summary": post_data.get("selftext", "")[:200],
"source": "Reddit",
"score": post_data.get("score", 0),
"fetched_at": datetime.now().isoformat()
})
# Sort by score descending
processed_articles.sort(key=lambda x: x["score"], reverse=True)
# Return as n8n items
return [{"json": article} for article in processed_articles]
```
### Key Techniques
- Process multiple data sources in one loop
- Normalize different data structures
- Use datetime for timestamps
- Sort by criteria
- Return properly formatted items
---
## Pattern 2: Regex-Based Filtering
**Use case**: Filter items based on pattern matching in text fields.
**Scenario**: Filter support tickets by priority keywords.
### Implementation
```python
import re
all_items = _input.all()
priority_tickets = []
# High priority keywords pattern
high_priority_pattern = re.compile(
r'\b(urgent|critical|emergency|asap|down|outage|broken)\b',
re.IGNORECASE
)
for item in all_items:
ticket = item["json"]
# Check subject and description
subject = ticket.get("subject", "")
description = ticket.get("description", "")
combined_text = f"{subject} {description}"
# Find matches
matches = high_priority_pattern.findall(combined_text)
if matches:
priority_tickets.append({
"json": {
**ticket,
"priority": "high",
"matched_keywords": list(set(matches)),
"keyword_count": len(matches)
}
})
else:
priority_tickets.append({
"json": {
**ticket,
"priority": "normal",
"matched_keywords": [],
"keyword_count": 0
}
})
# Sort by keyword count (most urgent first)
priority_tickets.sort(key=lambda x: x["json"]["keyword_count"], reverse=True)
return priority_tickets
```
### Key Techniques
- Use re.compile() for reusable patterns
- re.IGNORECASE for case-insensitive matching
- Combine multiple text fields for searching
- Extract and deduplicate matches
- Sort by priority indicators
---
## Pattern 3: Markdown to Structured Data
**Use case**: Parse markdown text into structured data.
**Scenario**: Extract tasks from markdown checklist.
### Implementation
```python
import re
markdown_text = _input.first()["json"]["body"].get("markdown", "")
# Parse markdown checklist
tasks = []
lines = markdown_text.split("\n")
for line in lines:
# Match: - [ ] Task or - [x] Task
match = re.match(r'^\s*-\s*\[([ x])\]\s*(.+)$', line, re.IGNORECASE)
if match:
checked = match.group(1).lower() == 'x'
task_text = match.group(2).strip()
# Extract priority if present (e.g., [P1], [HIGH])
priority_match = re.search(r'\[(P\d|HIGH|MEDIUM|LOW)\]', task_text, re.IGNORECASE)
priority = priority_match.group(1).upper() if priority_match else "NORMAL"
# Remove priority tag from text
clean_text = re.sub(r'\[(P\d|HIGH|MEDIUM|LOW)\]', '', task_text, flags=re.IGNORECASE).strip()
tasks.append({
"text": clean_text,
"completed": checked,
"priority": priority,
"original_line": line.strip()
})
return [{
"json": {
"tasks": tasks,
"total": len(tasks),
"completed": sum(1 for t in tasks if t["completed"]),
"pending": sum(1 for t in tasks if not t["completed"])
}
}]
```
### Key Techniques
- Line-by-line parsing
- Multiple regex patterns for extraction
- Extract metadata from text
- Calculate summary statistics
- Return structured data
---
## Pattern 4: JSON Object Comparison
**Use case**: Compare two JSON objects to find differences.
**Scenario**: Compare old and new user profile data.
### Implementation
```python
import json
all_items = _input.all()
# Assume first item is old data, second is new data
old_data = all_items[0]["json"] if len(all_items) > 0 else {}
new_data = all_items[1]["json"] if len(all_items) > 1 else {}
changes = {
"added": {},
"removed": {},
"modified": {},
"unchanged": {}
}
# Find all unique keys
all_keys = set(old_data.keys()) | set(new_data.keys())
for key in all_keys:
old_value = old_data.get(key)
new_value = new_data.get(key)
if key not in old_data:
# Added field
changes["added"][key] = new_value
elif key not in new_data:
# Removed field
changes["removed"][key] = old_value
elif old_value != new_value:
# Modified field
changes["modified"][key] = {
"old": old_value,
"new": new_value
}
else:
# Unchanged field
changes["unchanged"][key] = old_value
return [{
"json": {
"changes": changes,
"summary": {
"added_count": len(changes["added"]),
"removed_count": len(changes["removed"]),
"modified_count": len(changes["modified"]),
"unchanged_count": len(changes["unchanged"]),
"has_changes": len(changes["added"]) > 0 or len(changes["removed"]) > 0 or len(changes["modified"]) > 0
}
}
}]
```
### Key Techniques
- Set operations for key comparison
- Dictionary .get() for safe access
- Categorize changes by type
- Create summary statistics
- Return detailed comparison
---
## Pattern 5: CRM Data Transformation
**Use case**: Transform CRM data to standard format.
**Scenario**: Normalize data from different CRM systems.
### Implementation
```python
from datetime import datetime
import re
all_items = _input.all()
normalized_contacts = []
for item in all_items:
raw_contact = item["json"]
source = raw_contact.get("source", "unknown")
# Normalize email
email = raw_contact.get("email", "").lower().strip()
# Normalize phone (remove non-digits)
phone_raw = raw_contact.get("phone", "")
phone = re.sub(r'\D', '', phone_raw)
# Parse name
if "full_name" in raw_contact:
name_parts = raw_contact["full_name"].split(" ", 1)
first_name = name_parts[0] if len(name_parts) > 0 else ""
last_name = name_parts[1] if len(name_parts) > 1 else ""
else:
first_name = raw_contact.get("first_name", "")
last_name = raw_contact.get("last_name", "")
# Normalize status
status_raw = raw_contact.get("status", "").lower()
status = "active" if status_raw in ["active", "enabled", "true", "1"] else "inactive"
# Create normalized contact
normalized_contacts.append({
"json": {
"id": raw_contact.get("id", ""),
"first_name": first_name.strip(),
"last_name": last_name.strip(),
"full_name": f"{first_name} {last_name}".strip(),
"email": email,
"phone": phone,
"status": status,
"source": source,
"normalized_at": datetime.now().isoformat(),
"original_data": raw_contact
}
})
return normalized_contacts
```
### Key Techniques
- Multiple field name variations handling
- String cleaning and normalization
- Regex for phone number cleaning
- Name parsing logic
- Status normalization
- Preserve original data
---
## Pattern 6: Release Notes Processing
**Use case**: Parse release notes and categorize changes.
**Scenario**: Extract features, fixes, and breaking changes from release notes.
### Implementation
```python
import re
release_notes = _input.first()["json"]["body"].get("notes", "")
categories = {
"features": [],
"fixes": [],
"breaking": [],
"other": []
}
# Split into lines
lines = release_notes.split("\n")
for line in lines:
line = line.strip()
# Skip empty lines and headers
if not line or line.startswith("#"):
continue
# Remove bullet points
clean_line = re.sub(r'^[\*\-\+]\s*', '', line)
# Categorize
if re.search(r'\b(feature|add|new)\b', clean_line, re.IGNORECASE):
categories["features"].append(clean_line)
elif re.search(r'\b(fix|bug|patch|resolve)\b', clean_line, re.IGNORECASE):
categories["fixes"].append(clean_line)
elif re.search(r'\b(breaking|deprecated|remove)\b', clean_line, re.IGNORECASE):
categories["breaking"].append(clean_line)
else:
categories["other"].append(clean_line)
return [{
"json": {
"categories": categories,
"summary": {
"features": len(categories["features"]),
"fixes": len(categories["fixes"]),
"breaking": len(categories["breaking"]),
"other": len(categories["other"]),
"total": sum(len(v) for v in categories.values())
}
}
}]
```
### Key Techniques
- Line-by-line parsing
- Pattern-based categorization
- Bullet point removal
- Skip headers and empty lines
- Summary statistics
---
## Pattern 7: Array Transformation
**Use case**: Reshape arrays and extract specific fields.
**Scenario**: Transform user data array to extract specific fields.
### Implementation
```python
all_items = _input.all()
# Extract and transform
transformed = []
for item in all_items:
user = item["json"]
# Extract nested fields
profile = user.get("profile", {})
settings = user.get("settings", {})
transformed.append({
"json": {
"user_id": user.get("id"),
"email": user.get("email"),
"name": profile.get("name", "Unknown"),
"avatar": profile.get("avatar_url"),
"bio": profile.get("bio", "")[:100], # Truncate to 100 chars
"notifications_enabled": settings.get("notifications", True),
"theme": settings.get("theme", "light"),
"created_at": user.get("created_at"),
"last_login": user.get("last_login_at")
}
})
return transformed
```
### Key Techniques
- Field extraction from nested objects
- Default values with .get()
- String truncation
- Flattening nested structures
---
## Pattern 8: Dictionary Lookup
**Use case**: Create lookup dictionary for fast data access.
**Scenario**: Look up user details by ID.
### Implementation
```python
all_items = _input.all()
# Build lookup dictionary
users_by_id = {}
for item in all_items:
user = item["json"]
user_id = user.get("id")
if user_id:
users_by_id[user_id] = {
"name": user.get("name"),
"email": user.get("email"),
"status": user.get("status")
}
# Example: Look up specific users
lookup_ids = [1, 3, 5]
looked_up = []
for user_id in lookup_ids:
if user_id in users_by_id:
looked_up.append({
"json": {
"id": user_id,
**users_by_id[user_id],
"found": True
}
})
else:
looked_up.append({
"json": {
"id": user_id,
"found": False
}
})
return looked_up
```
### Key Techniques
- Dictionary comprehension alternative
- O(1) lookup time
- Handle missing keys gracefully
- Preserve lookup order
---
## Pattern 9: Top N Filtering
**Use case**: Get top items by score or value.
**Scenario**: Get top 10 products by sales.
### Implementation
```python
all_items = _input.all()
# Extract products with sales
products = []
for item in all_items:
product = item["json"]
products.append({
"id": product.get("id"),
"name": product.get("name"),
"sales": product.get("sales", 0),
"revenue": product.get("revenue", 0.0),
"category": product.get("category")
})
# Sort by sales descending
products.sort(key=lambda p: p["sales"], reverse=True)
# Get top 10
top_10 = products[:10]
return [
{
"json": {
**product,
"rank": index + 1
}
}
for index, product in enumerate(top_10)
]
```
### Key Techniques
- List sorting with custom key
- Slicing for top N
- Add ranking information
- Enumerate for index
---
## Pattern 10: String Aggregation
**Use case**: Aggregate strings with formatting.
**Scenario**: Create summary text from multiple items.
### Implementation
```python
all_items = _input.all()
# Collect messages
messages = []
for item in all_items:
data = item["json"]
user = data.get("user", "Unknown")
message = data.get("message", "")
timestamp = data.get("timestamp", "")
# Format each message
formatted = f"[{timestamp}] {user}: {message}"
messages.append(formatted)
# Join with newlines
summary = "\n".join(messages)
# Create statistics
total_length = sum(len(msg) for msg in messages)
average_length = total_length / len(messages) if messages else 0
return [{
"json": {
"summary": summary,
"message_count": len(messages),
"total_characters": total_length,
"average_length": round(average_length, 2)
}
}]
```
### Key Techniques
- String formatting with f-strings
- Join lists with separator
- Calculate string statistics
- Handle empty lists
---
## Pattern Comparison: Python vs JavaScript
### Data Access
```python
# Python
all_items = _input.all()
first_item = _input.first()
current = _input.item
webhook_data = _json["body"]
# JavaScript
const allItems = $input.all();
const firstItem = $input.first();
const current = $input.item;
const webhookData = $json.body;
```
### Dictionary/Object Access
```python
# Python - Dictionary key access
name = user["name"] # May raise KeyError
name = user.get("name", "?") # Safe with default
# JavaScript - Object property access
const name = user.name; // May be undefined
const name = user.name || "?"; // Safe with default
```
### Array Operations
```python
# Python - List comprehension
filtered = [item for item in items if item["active"]]
# JavaScript - Array methods
const filtered = items.filter(item => item.active);
```
### Sorting
```python
# Python
items.sort(key=lambda x: x["score"], reverse=True)
# JavaScript
items.sort((a, b) => b.score - a.score);
```
---
## Best Practices
### 1. Use .get() for Safe Access
```python
# ✅ SAFE: Use .get() with defaults
name = user.get("name", "Unknown")
email = user.get("email", "no-email@example.com")
# ❌ RISKY: Direct key access
name = user["name"] # KeyError if missing!
```
### 2. Handle Empty Lists
```python
# ✅ SAFE: Check before processing
items = _input.all()
if items:
first = items[0]
else:
return [{"json": {"error": "No items"}}]
# ❌ RISKY: Assume items exist
first = items[0] # IndexError if empty!
```
### 3. Use List Comprehensions
```python
# ✅ PYTHONIC: List comprehension
active = [item for item in items if item["json"].get("active")]
# ❌ VERBOSE: Traditional loop
active = []
for item in items:
if item["json"].get("active"):
active.append(item)
```
### 4. Return Proper Format
```python
# ✅ CORRECT: Array of objects with "json" key
return [{"json": {"field": "value"}}]
# ❌ WRONG: Just the data
return {"field": "value"}
# ❌ WRONG: Array without "json" wrapper
return [{"field": "value"}]
```
### 5. Use Standard Library
```python
# ✅ GOOD: Use standard library
import statistics
average = statistics.mean(numbers)
# ✅ ALSO GOOD: Built-in functions
average = sum(numbers) / len(numbers) if numbers else 0
# ❌ CAN'T DO: External libraries
import numpy as np # ModuleNotFoundError!
```
---
## When to Use Each Pattern
| Pattern | When to Use |
|---------|-------------|
| Multi-Source Aggregation | Combining data from different nodes/sources |
| Regex Filtering | Text pattern matching, validation, extraction |
| Markdown Parsing | Processing formatted text into structured data |
| JSON Comparison | Detecting changes between objects |
| CRM Transformation | Normalizing data from different systems |
| Release Notes | Categorizing text by keywords |
| Array Transformation | Reshaping data, extracting fields |
| Dictionary Lookup | Fast ID-based lookups |
| Top N Filtering | Getting best/worst items by criteria |
| String Aggregation | Creating formatted text summaries |
---
## Summary
**Key Takeaways**:
- Use `.get()` for safe dictionary access
- List comprehensions are pythonic and efficient
- Handle empty lists/None values
- Use standard library (json, datetime, re)
- Return proper n8n format: `[{"json": {...}}]`
**Remember**:
- JavaScript is recommended for 95% of use cases
- Python has NO external libraries
- Use n8n nodes for complex operations
- Code node is for data transformation, not API calls
**See Also**:
- [SKILL.md](SKILL.md) - Python Code overview
- [DATA_ACCESS.md](DATA_ACCESS.md) - Data access patterns
- [STANDARD_LIBRARY.md](STANDARD_LIBRARY.md) - Available modules
- [ERROR_PATTERNS.md](ERROR_PATTERNS.md) - Avoid common mistakes

702
.opencode/skills-off/n8n-code-python/DATA_ACCESS.md Normal file
View File

@@ -0,0 +1,702 @@
# Data Access Patterns - Python Code Node
Complete guide to accessing data in n8n Code nodes using Python.
---
## Overview
In n8n Python Code nodes, you access data using **underscore-prefixed** variables: `_input`, `_json`, `_node`.
**Data Access Priority** (by common usage):
1. **`_input.all()`** - Most common - Batch operations, aggregations
2. **`_input.first()`** - Very common - Single item operations
3. **`_input.item`** - Common - Each Item mode only
4. **`_node["NodeName"]["json"]`** - Specific node references
5. **`_json`** - Direct current item (use `_input` instead)
**Python vs JavaScript**:
| JavaScript | Python (Beta) | Python (Native) |
|------------|---------------|-----------------|
| `$input.all()` | `_input.all()` | `_items` |
| `$input.first()` | `_input.first()` | `_items[0]` |
| `$input.item` | `_input.item` | `_item` |
| `$json` | `_json` | `_item["json"]` |
| `$node["Name"]` | `_node["Name"]` | Not available |
---
## Pattern 1: _input.all() - Process All Items
**Usage**: Most common pattern for batch processing
**When to use:**
- Processing multiple records
- Aggregating data (sum, count, average)
- Filtering lists
- Transforming datasets
### Basic Usage
```python
# Get all items from previous node
all_items = _input.all()
# all_items is a list of dictionaries like:
# [
# {"json": {"id": 1, "name": "Alice"}},
# {"json": {"id": 2, "name": "Bob"}}
# ]
print(f"Received {len(all_items)} items")
return all_items
```
### Example 1: Filter Active Items
```python
all_items = _input.all()
# Filter only active items
active_items = [
item for item in all_items
if item["json"].get("status") == "active"
]
return active_items
```
### Example 2: Transform All Items
```python
all_items = _input.all()
# Transform to new structure
transformed = []
for item in all_items:
transformed.append({
"json": {
"id": item["json"].get("id"),
"full_name": f"{item['json'].get('first_name', '')} {item['json'].get('last_name', '')}",
"email": item["json"].get("email"),
"processed_at": datetime.now().isoformat()
}
})
return transformed
```
### Example 3: Aggregate Data
```python
all_items = _input.all()
# Calculate total
total = sum(item["json"].get("amount", 0) for item in all_items)
return [{
"json": {
"total": total,
"count": len(all_items),
"average": total / len(all_items) if all_items else 0
}
}]
```
### Example 4: Sort and Limit
```python
all_items = _input.all()
# Get top 5 by score
sorted_items = sorted(
all_items,
key=lambda item: item["json"].get("score", 0),
reverse=True
)
top_five = sorted_items[:5]
return [{"json": item["json"]} for item in top_five]
```
### Example 5: Group By Category
```python
all_items = _input.all()
# Group items by category
grouped = {}
for item in all_items:
category = item["json"].get("category", "Uncategorized")
if category not in grouped:
grouped[category] = []
grouped[category].append(item["json"])
# Convert to list format
return [
{
"json": {
"category": category,
"items": items,
"count": len(items)
}
}
for category, items in grouped.items()
]
```
### Example 6: Deduplicate by ID
```python
all_items = _input.all()
# Remove duplicates by ID
seen = set()
unique = []
for item in all_items:
item_id = item["json"].get("id")
if item_id and item_id not in seen:
seen.add(item_id)
unique.append(item)
return unique
```
---
## Pattern 2: _input.first() - Get First Item
**Usage**: Very common for single-item operations
**When to use:**
- Previous node returns single object
- Working with API responses
- Getting initial/first data point
### Basic Usage
```python
# Get first item from previous node
first_item = _input.first()
# Access the JSON data
data = first_item["json"]
print(f"First item: {data}")
return [{"json": data}]
```
### Example 1: Process Single API Response
```python
# Get API response (typically single object)
response = _input.first()["json"]
# Extract what you need
return [{
"json": {
"user_id": response.get("data", {}).get("user", {}).get("id"),
"user_name": response.get("data", {}).get("user", {}).get("name"),
"status": response.get("status"),
"fetched_at": datetime.now().isoformat()
}
}]
```
### Example 2: Transform Single Object
```python
data = _input.first()["json"]
# Transform structure
return [{
"json": {
"id": data.get("id"),
"contact": {
"email": data.get("email"),
"phone": data.get("phone")
},
"address": {
"street": data.get("street"),
"city": data.get("city"),
"zip": data.get("zip")
}
}
}]
```
### Example 3: Validate Single Item
```python
item = _input.first()["json"]
# Validation logic
is_valid = bool(item.get("email") and "@" in item.get("email", ""))
return [{
"json": {
**item,
"valid": is_valid,
"validated_at": datetime.now().isoformat()
}
}]
```
### Example 4: Extract Nested Data
```python
response = _input.first()["json"]
# Navigate nested structure
users = response.get("data", {}).get("users", [])
return [
{
"json": {
"id": user.get("id"),
"name": user.get("profile", {}).get("name", "Unknown"),
"email": user.get("contact", {}).get("email", "no-email")
}
}
for user in users
]
```
---
## Pattern 3: _input.item - Current Item (Each Item Mode)
**Usage**: Common in "Run Once for Each Item" mode
**When to use:**
- Mode is set to "Run Once for Each Item"
- Need to process items independently
- Per-item API calls or validations
**IMPORTANT**: Only use in "Each Item" mode. Will be undefined in "All Items" mode.
### Basic Usage
```python
# In "Run Once for Each Item" mode
current_item = _input.item
data = current_item["json"]
print(f"Processing item: {data.get('id')}")
return [{
"json": {
**data,
"processed": True
}
}]
```
### Example 1: Add Processing Metadata
```python
item = _input.item
return [{
"json": {
**item["json"],
"processed": True,
"processed_at": datetime.now().isoformat(),
"processing_duration": random.random() * 1000 # Simulated
}
}]
```
### Example 2: Per-Item Validation
```python
item = _input.item
data = item["json"]
# Validate this specific item
errors = []
if not data.get("email"):
errors.append("Email required")
if not data.get("name"):
errors.append("Name required")
if data.get("age") and data["age"] < 18:
errors.append("Must be 18+")
return [{
"json": {
**data,
"valid": len(errors) == 0,
"errors": errors if errors else None
}
}]
```
### Example 3: Conditional Processing
```python
item = _input.item
data = item["json"]
# Process based on item type
if data.get("type") == "premium":
return [{
"json": {
**data,
"discount": 0.20,
"tier": "premium"
}
}]
else:
return [{
"json": {
**data,
"discount": 0.05,
"tier": "standard"
}
}]
```
---
## Pattern 4: _node - Reference Other Nodes
**Usage**: Less common, but powerful for specific scenarios
**When to use:**
- Need data from specific named node
- Combining data from multiple nodes
### Basic Usage
```python
# Get output from specific node
webhook_data = _node["Webhook"]["json"]
api_data = _node["HTTP Request"]["json"]
return [{
"json": {
"from_webhook": webhook_data,
"from_api": api_data
}
}]
```
### Example 1: Combine Multiple Sources
```python
# Reference multiple nodes
webhook = _node["Webhook"]["json"]
database = _node["Postgres"]["json"]
api = _node["HTTP Request"]["json"]
return [{
"json": {
"combined": {
"webhook": webhook.get("body", {}),
"db_records": len(database) if isinstance(database, list) else 1,
"api_response": api.get("status")
},
"processed_at": datetime.now().isoformat()
}
}]
```
### Example 2: Compare Across Nodes
```python
old_data = _node["Get Old Data"]["json"]
new_data = _node["Get New Data"]["json"]
# Simple comparison
changes = {
"added": [n for n in new_data if n.get("id") not in [o.get("id") for o in old_data]],
"removed": [o for o in old_data if o.get("id") not in [n.get("id") for n in new_data]]
}
return [{
"json": {
"changes": changes,
"summary": {
"added": len(changes["added"]),
"removed": len(changes["removed"])
}
}
}]
```
---
## Critical: Webhook Data Structure
**MOST COMMON MISTAKE**: Forgetting webhook data is nested under `["body"]`
### The Problem
Webhook node wraps all incoming data under a `"body"` property.
### Structure
```python
# Webhook node output structure:
{
"headers": {
"content-type": "application/json",
"user-agent": "..."
},
"params": {},
"query": {},
"body": {
# ← YOUR DATA IS HERE
"name": "Alice",
"email": "alice@example.com",
"message": "Hello!"
}
}
```
### Wrong vs Right
```python
# ❌ WRONG: Trying to access directly
name = _json["name"] # KeyError!
email = _json["email"] # KeyError!
# ✅ CORRECT: Access via ["body"]
name = _json["body"]["name"] # "Alice"
email = _json["body"]["email"] # "alice@example.com"
# ✅ SAFER: Use .get() for safe access
webhook_data = _json.get("body", {})
name = webhook_data.get("name") # None if missing
email = webhook_data.get("email", "no-email") # Default value
```
### Example: Full Webhook Processing
```python
# Get webhook data from previous node
webhook_output = _input.first()["json"]
# Access the actual payload
payload = webhook_output.get("body", {})
# Access headers if needed
content_type = webhook_output.get("headers", {}).get("content-type")
# Access query parameters if needed
api_key = webhook_output.get("query", {}).get("api_key")
# Process the actual data
return [{
"json": {
# Data from webhook body
"user_name": payload.get("name"),
"user_email": payload.get("email"),
"message": payload.get("message"),
# Metadata
"received_at": datetime.now().isoformat(),
"content_type": content_type,
"authenticated": bool(api_key)
}
}]
```
### POST Data, Query Params, and Headers
```python
webhook = _input.first()["json"]
return [{
"json": {
# POST body data
"form_data": webhook.get("body", {}),
# Query parameters (?key=value)
"query_params": webhook.get("query", {}),
# HTTP headers
"user_agent": webhook.get("headers", {}).get("user-agent"),
"content_type": webhook.get("headers", {}).get("content-type"),
# Request metadata
"method": webhook.get("method"), # POST, GET, etc.
"url": webhook.get("url")
}
}]
```
---
## Choosing the Right Pattern
### Decision Tree
```
Do you need ALL items from previous node?
├─ YES → Use _input.all()
│
└─ NO → Do you need just the FIRST item?
├─ YES → Use _input.first()
│
└─ NO → Are you in "Each Item" mode?
├─ YES → Use _input.item
│
└─ NO → Do you need specific node data?
├─ YES → Use _node["NodeName"]
└─ NO → Use _input.first() (default)
```
### Quick Reference Table
| Scenario | Use This | Example |
|----------|----------|---------|
| Sum all amounts | `_input.all()` | `sum(i["json"].get("amount", 0) for i in items)` |
| Get API response | `_input.first()` | `_input.first()["json"].get("data")` |
| Process each independently | `_input.item` | `_input.item["json"]` (Each Item mode) |
| Combine two nodes | `_node["Name"]` | `_node["API"]["json"]` |
| Filter list | `_input.all()` | `[i for i in items if i["json"].get("active")]` |
| Transform single object | `_input.first()` | `{**_input.first()["json"], "new": True}` |
| Webhook data | `_input.first()` | `_input.first()["json"]["body"]` |
---
## Common Mistakes
### Mistake 1: Using _json Without Context
```python
# ❌ RISKY: _json is ambiguous
value = _json["field"]
# ✅ CLEAR: Be explicit
value = _input.first()["json"]["field"]
```
### Mistake 2: Forgetting ["json"] Property
```python
# ❌ WRONG: Trying to access fields on item dictionary
items = _input.all()
names = [item["name"] for item in items] # KeyError!
# ✅ CORRECT: Access via ["json"]
names = [item["json"]["name"] for item in items]
```
### Mistake 3: Using _input.item in All Items Mode
```python
# ❌ WRONG: _input.item is None in "All Items" mode
data = _input.item["json"] # AttributeError!
# ✅ CORRECT: Use appropriate method
data = _input.first()["json"] # Or _input.all()
```
### Mistake 4: Not Handling Empty Lists
```python
# ❌ WRONG: Crashes if no items
first = _input.all()[0]["json"] # IndexError!
# ✅ CORRECT: Check length first
items = _input.all()
if items:
first = items[0]["json"]
else:
return []
# ✅ ALSO CORRECT: Use _input.first()
first = _input.first()["json"] # Built-in safety
```
### Mistake 5: Direct Dictionary Access (KeyError)
```python
# ❌ RISKY: Crashes if key missing
value = item["json"]["field"] # KeyError!
# ✅ SAFE: Use .get()
value = item["json"].get("field", "default")
```
---
## Advanced Patterns
### Pattern: Safe Nested Access
```python
# Deep nested access with .get()
value = (
_input.first()["json"]
.get("level1", {})
.get("level2", {})
.get("level3", "default")
)
```
### Pattern: List Comprehension with Filtering
```python
items = _input.all()
# Filter and transform in one step
result = [
{
"json": {
"id": item["json"]["id"],
"name": item["json"]["name"].upper()
}
}
for item in items
if item["json"].get("active") and item["json"].get("verified")
]
return result
```
### Pattern: Dictionary Comprehension
```python
items = _input.all()
# Create lookup dictionary
lookup = {
item["json"]["id"]: item["json"]
for item in items
if "id" in item["json"]
}
return [{"json": lookup}]
```
---
## Summary
**Most Common Patterns**:
1. `_input.all()` - Process multiple items, batch operations
2. `_input.first()` - Single item, API responses
3. `_input.item` - Each Item mode processing
**Critical Rule**:
- Webhook data is under `["body"]` property
**Best Practice**:
- Use `.get()` for dictionary access to avoid KeyError
- Always check for empty lists
- Be explicit: Use `_input.first()["json"]["field"]` instead of `_json["field"]`
**See Also**:
- [SKILL.md](SKILL.md) - Overview and quick start
- [COMMON_PATTERNS.md](COMMON_PATTERNS.md) - Python-specific patterns
- [ERROR_PATTERNS.md](ERROR_PATTERNS.md) - Avoid common mistakes

601
.opencode/skills-off/n8n-code-python/ERROR_PATTERNS.md Normal file
View File

@@ -0,0 +1,601 @@
# Error Patterns - Python Code Node
Common Python Code node errors and how to fix them.
---
## Error Overview
**Top 5 Python Code Node Errors**:
1. **ModuleNotFoundError** - Trying to import external libraries (Python-specific)
2. **Empty Code / Missing Return** - No code or return statement
3. **KeyError** - Dictionary access without .get()
4. **IndexError** - List access without bounds checking
5. **Incorrect Return Format** - Wrong data structure returned
These 5 errors cover the majority of Python Code node failures.
---
## Error #1: ModuleNotFoundError (MOST CRITICAL)
**Frequency**: Very common in Python Code nodes
**What it is**: Attempting to import external libraries that aren't available.
### The Problem
```python
# ❌ WRONG: External libraries not available
import requests # ModuleNotFoundError: No module named 'requests'
import pandas # ModuleNotFoundError: No module named 'pandas'
import numpy # ModuleNotFoundError: No module named 'numpy'
import bs4 # ModuleNotFoundError: No module named 'bs4'
import pymongo # ModuleNotFoundError: No module named 'pymongo'
import psycopg2 # ModuleNotFoundError: No module named 'psycopg2'
# This code will FAIL - these libraries are not installed!
response = requests.get("https://api.example.com/data")
```
### The Solution
**Option 1: Use JavaScript Instead** (Recommended for 95% of cases)
```javascript
// ✅ JavaScript Code node with $helpers.httpRequest()
const response = await $helpers.httpRequest({
method: 'GET',
url: 'https://api.example.com/data'
});
return [{json: response}];
```
**Option 2: Use n8n HTTP Request Node**
```python
# ✅ Add HTTP Request node BEFORE Python Code node
# Access the response in Python Code node
response = _input.first()["json"]
return [{
"json": {
"status": response.get("status"),
"data": response.get("body"),
"processed": True
}
}]
```
**Option 3: Use Standard Library Only**
```python
# ✅ Use urllib from standard library (limited functionality)
from urllib.request import urlopen
from urllib.parse import urlencode
import json
# Simple GET request (no headers, no auth)
url = "https://api.example.com/data"
with urlopen(url) as response:
data = json.loads(response.read())
return [{"json": data}]
```
### Common Library Replacements
| Need | ❌ External Library | ✅ Alternative |
|------|-------------------|----------------|
| HTTP requests | `requests` | Use HTTP Request node or JavaScript |
| Data analysis | `pandas` | Use Python list comprehensions |
| Database | `psycopg2`, `pymongo` | Use n8n database nodes |
| Web scraping | `beautifulsoup4` | Use HTML Extract node |
| Excel | `openpyxl` | Use Spreadsheet File node |
| Image processing | `pillow` | Use external API or node |
### Available Standard Library Modules
```python
# ✅ THESE WORK - Standard library only
import json # JSON parsing
import datetime # Date/time operations
import re # Regular expressions
import base64 # Base64 encoding
import hashlib # Hashing (MD5, SHA256)
import urllib.parse # URL parsing and encoding
import math # Math functions
import random # Random numbers
import statistics # Statistical functions
import collections # defaultdict, Counter, etc.
```
---
## Error #2: Empty Code / Missing Return
**Frequency**: Common across all Code nodes
**What it is**: Code node has no code or no return statement.
### The Problem
```python
# ❌ WRONG: Empty code
# (nothing here)
# ❌ WRONG: Code but no return
items = _input.all()
processed = [item for item in items if item["json"].get("active")]
# Forgot to return!
# ❌ WRONG: Return in wrong scope
if _input.all():
return [{"json": {"result": "success"}}]
# Return is inside if block - may not execute!
```
### The Solution
```python
# ✅ CORRECT: Always return
all_items = _input.all()
if not all_items:
# Return empty array or error
return [{"json": {"error": "No items"}}]
# Process items
processed = [item for item in all_items if item["json"].get("active")]
# Always return at the end
return processed if processed else [{"json": {"message": "No active items"}}]
```
### Best Practice
```python
# ✅ GOOD: Return at end of function (unconditional)
def process_items():
items = _input.all()
if not items:
return [{"json": {"error": "Empty input"}}]
# Process
result = []
for item in items:
result.append({"json": item["json"]})
return result
# Call function and return result
return process_items()
```
---
## Error #3: KeyError
**Frequency**: Very common in Python Code nodes
**What it is**: Accessing dictionary key that doesn't exist.
### The Problem
```python
# ❌ WRONG: Direct key access
item = _input.first()["json"]
name = item["name"] # KeyError if "name" doesn't exist!
email = item["email"] # KeyError if "email" doesn't exist!
age = item["age"] # KeyError if "age" doesn't exist!
return [{
"json": {
"name": name,
"email": email,
"age": age
}
}]
```
### Error Message
```
KeyError: 'name'
```
### The Solution
```python
# ✅ CORRECT: Use .get() with defaults
item = _input.first()["json"]
name = item.get("name", "Unknown")
email = item.get("email", "no-email@example.com")
age = item.get("age", 0)
return [{
"json": {
"name": name,
"email": email,
"age": age
}
}]
```
### Nested Dictionary Access
```python
# ❌ WRONG: Nested key access
webhook = _input.first()["json"]
name = webhook["body"]["user"]["name"] # Multiple KeyErrors possible!
# ✅ CORRECT: Safe nested access
webhook = _input.first()["json"]
body = webhook.get("body", {})
user = body.get("user", {})
name = user.get("name", "Unknown")
# ✅ ALSO CORRECT: Chained .get()
name = (
webhook
.get("body", {})
.get("user", {})
.get("name", "Unknown")
)
return [{"json": {"name": name}}]
```
### Webhook Body Access (Critical!)
```python
# ❌ WRONG: Forgetting webhook data is under "body"
webhook = _input.first()["json"]
name = webhook["name"] # KeyError!
email = webhook["email"] # KeyError!
# ✅ CORRECT: Access via ["body"]
webhook = _input.first()["json"]
body = webhook.get("body", {})
name = body.get("name", "Unknown")
email = body.get("email", "no-email")
return [{
"json": {
"name": name,
"email": email
}
}]
```
---
## Error #4: IndexError
**Frequency**: Common when processing arrays/lists
**What it is**: Accessing list index that doesn't exist.
### The Problem
```python
# ❌ WRONG: Assuming items exist
all_items = _input.all()
first_item = all_items[0] # IndexError if list is empty!
second_item = all_items[1] # IndexError if only 1 item!
return [{
"json": {
"first": first_item["json"],
"second": second_item["json"]
}
}]
```
### Error Message
```
IndexError: list index out of range
```
### The Solution
```python
# ✅ CORRECT: Check length first
all_items = _input.all()
if len(all_items) >= 2:
first_item = all_items[0]["json"]
second_item = all_items[1]["json"]
return [{
"json": {
"first": first_item,
"second": second_item
}
}]
else:
return [{
"json": {
"error": f"Expected 2+ items, got {len(all_items)}"
}
}]
```
### Safe First Item Access
```python
# ✅ CORRECT: Use _input.first() instead of [0]
# This is safer than manual indexing
first_item = _input.first()["json"]
return [{"json": first_item}]
# ✅ ALSO CORRECT: Check before accessing
all_items = _input.all()
if all_items:
first_item = all_items[0]["json"]
else:
first_item = {}
return [{"json": first_item}]
```
### Slice Instead of Index
```python
# ✅ CORRECT: Use slicing (never raises IndexError)
all_items = _input.all()
# Get first 5 items (won't fail if fewer than 5)
first_five = all_items[:5]
# Get items after first (won't fail if empty)
rest = all_items[1:]
return [{"json": item["json"]} for item in first_five]
```
---
## Error #5: Incorrect Return Format
**Frequency**: Common for new users
**What it is**: Returning data in wrong format (n8n expects array of objects with "json" key).
### The Problem
```python
# ❌ WRONG: Returning plain dictionary
return {"name": "Alice", "age": 30}
# ❌ WRONG: Returning array without "json" wrapper
return [{"name": "Alice"}, {"name": "Bob"}]
# ❌ WRONG: Returning None
return None
# ❌ WRONG: Returning string
return "success"
# ❌ WRONG: Returning single item (not array)
return {"json": {"name": "Alice"}}
```
### The Solution
```python
# ✅ CORRECT: Array of objects with "json" key
return [{"json": {"name": "Alice", "age": 30}}]
# ✅ CORRECT: Multiple items
return [
{"json": {"name": "Alice"}},
{"json": {"name": "Bob"}}
]
# ✅ CORRECT: Transform items
all_items = _input.all()
return [
{"json": item["json"]}
for item in all_items
]
# ✅ CORRECT: Empty array (valid)
return []
# ✅ CORRECT: Single item still needs array wrapper
return [{"json": {"result": "success"}}]
```
### Common Scenarios
**Scenario 1: Aggregation (Return Single Result)**
```python
# Calculate total
all_items = _input.all()
total = sum(item["json"].get("amount", 0) for item in all_items)
# ✅ CORRECT: Wrap in array with "json"
return [{
"json": {
"total": total,
"count": len(all_items)
}
}]
```
**Scenario 2: Filtering (Return Multiple Results)**
```python
# Filter active items
all_items = _input.all()
active = [item for item in all_items if item["json"].get("active")]
# ✅ CORRECT: Already in correct format
return active
# ✅ ALSO CORRECT: If transforming
return [
{"json": {**item["json"], "filtered": True}}
for item in active
]
```
**Scenario 3: No Results**
```python
# ✅ CORRECT: Return empty array
return []
# ✅ ALSO CORRECT: Return error message
return [{"json": {"error": "No results found"}}]
```
---
## Bonus Error: AttributeError
**What it is**: Using _input.item in wrong mode.
### The Problem
```python
# ❌ WRONG: Using _input.item in "All Items" mode
current = _input.item # None in "All Items" mode
data = current["json"] # AttributeError: 'NoneType' object has no attribute '__getitem__'
```
### The Solution
```python
# ✅ CORRECT: Check mode or use appropriate method
# In "All Items" mode, use:
all_items = _input.all()
# In "Each Item" mode, use:
current_item = _input.item
# ✅ SAFE: Check if item exists
current = _input.item
if current:
data = current["json"]
return [{"json": data}]
else:
# Running in "All Items" mode
return _input.all()
```
---
## Error Prevention Checklist
Before running your Python Code node, verify:
- [ ] **No external imports**: Only standard library (json, datetime, re, etc.)
- [ ] **Code returns data**: Every code path ends with `return`
- [ ] **Correct format**: Returns `[{"json": {...}}]` (array with "json" key)
- [ ] **Safe dictionary access**: Uses `.get()` instead of `[]` for dictionaries
- [ ] **Safe list access**: Checks length before indexing or uses slicing
- [ ] **Webhook body access**: Accesses webhook data via `_json["body"]`
- [ ] **No None returns**: Returns empty array `[]` instead of `None`
- [ ] **Mode awareness**: Uses `_input.all()`, `_input.first()`, or `_input.item` appropriately
---
## Quick Fix Reference
| Error | Quick Fix |
|-------|-----------|
| `ModuleNotFoundError` | Use JavaScript or HTTP Request node instead |
| `KeyError: 'field'` | Change `data["field"]` to `data.get("field", default)` |
| `IndexError: list index out of range` | Check `if len(items) > 0:` before `items[0]` |
| Empty output | Add `return [{"json": {...}}]` at end |
| `AttributeError: 'NoneType'` | Check mode setting or verify `_input.item` exists |
| Wrong format error | Wrap result: `return [{"json": result}]` |
| Webhook KeyError | Access via `_json.get("body", {})` |
---
## Testing Your Code
### Test Pattern 1: Handle Empty Input
```python
# ✅ Always test with empty input
all_items = _input.all()
if not all_items:
return [{"json": {"message": "No items to process"}}]
# Continue with processing
# ...
```
### Test Pattern 2: Test with Missing Fields
```python
# ✅ Use .get() with defaults
item = _input.first()["json"]
# These won't fail even if fields missing
name = item.get("name", "Unknown")
email = item.get("email", "no-email")
age = item.get("age", 0)
return [{"json": {"name": name, "email": email, "age": age}}]
```
### Test Pattern 3: Test Both Modes
```python
# ✅ Code that works in both modes
try:
# Try "Each Item" mode first
current = _input.item
if current:
return [{"json": current["json"]}]
except:
pass
# Fall back to "All Items" mode
all_items = _input.all()
return all_items if all_items else [{"json": {"message": "No data"}}]
```
---
## Summary
**Top 5 Errors to Avoid**:
1. **ModuleNotFoundError** - Use JavaScript or n8n nodes instead
2. **Missing return** - Always end with `return [{"json": {...}}]`
3. **KeyError** - Use `.get()` for dictionary access
4. **IndexError** - Check length before indexing
5. **Wrong format** - Return `[{"json": {...}}]`, not plain objects
**Golden Rules**:
- NO external libraries (use JavaScript instead)
- ALWAYS use `.get()` for dictionaries
- ALWAYS return `[{"json": {...}}]` format
- CHECK lengths before list access
- ACCESS webhook data via `["body"]`
**Remember**:
- JavaScript is recommended for 95% of use cases
- Python has limitations (no requests, pandas, numpy)
- Use n8n nodes for complex operations
**See Also**:
- [SKILL.md](SKILL.md) - Python Code overview
- [DATA_ACCESS.md](DATA_ACCESS.md) - Data access patterns
- [STANDARD_LIBRARY.md](STANDARD_LIBRARY.md) - Available modules
- [COMMON_PATTERNS.md](COMMON_PATTERNS.md) - Production patterns

386
.opencode/skills-off/n8n-code-python/README.md Normal file
View File

@@ -0,0 +1,386 @@
# n8n Code Python Skill
Expert guidance for writing Python code in n8n Code nodes.
---
## ⚠️ Important: JavaScript First
**Use JavaScript for 95% of use cases.**
Python in n8n has **NO external libraries** (no requests, pandas, numpy).
**When to use Python**:
- You have complex Python-specific logic
- You need Python's standard library features
- You're more comfortable with Python than JavaScript
**When to use JavaScript** (recommended):
- HTTP requests ($helpers.httpRequest available)
- Date/time operations (Luxon library included)
- Most data transformations
- When in doubt
---
## What This Skill Teaches
### Core Concepts
1. **Critical Limitation**: No external libraries
2. **Data Access**: `_input.all()`, `_input.first()`, `_input.item`
3. **Webhook Gotcha**: Data is under `_json["body"]`
4. **Return Format**: Must return `[{"json": {...}}]`
5. **Standard Library**: json, datetime, re, base64, hashlib, etc.
### Top 5 Error Prevention
This skill emphasizes **error prevention**:
1. **ModuleNotFoundError** (trying to import external libraries)
2. **Empty code / missing return**
3. **KeyError** (dictionary access without .get())
4. **IndexError** (list access without bounds checking)
5. **Incorrect return format**
These 5 errors are the most common in Python Code nodes.
---
## Skill Activation
This skill activates when you:
- Write Python in Code nodes
- Ask about Python limitations
- Need to know available standard library
- Troubleshoot Python Code node errors
- Work with Python data structures
**Example queries**:
- "Can I use pandas in Python Code node?"
- "How do I access webhook data in Python?"
- "What Python libraries are available?"
- "Write Python code to process JSON"
- "Why is requests module not found?"
---
## File Structure
### SKILL.md (719 lines)
**Quick start** and overview
- When to use Python vs JavaScript
- Critical limitation (no external libraries)
- Mode selection (All Items vs Each Item)
- Data access overview
- Return format requirements
- Standard library overview
### DATA_ACCESS.md (703 lines)
**Complete data access patterns**
- `_input.all()` - Process all items
- `_input.first()` - Get first item
- `_input.item` - Current item (Each Item mode)
- `_node["Name"]` - Reference other nodes
- Webhook body structure (critical gotcha!)
- Pattern selection guide
### STANDARD_LIBRARY.md (850 lines)
**Available Python modules**
- json - JSON parsing
- datetime - Date/time operations
- re - Regular expressions
- base64 - Encoding/decoding
- hashlib - Hashing
- urllib.parse - URL operations
- math, random, statistics
- What's NOT available (requests, pandas, numpy)
- Workarounds for missing libraries
### COMMON_PATTERNS.md (895 lines)
**10 production-tested patterns**
1. Multi-source data aggregation
2. Regex-based filtering
3. Markdown to structured data
4. JSON object comparison
5. CRM data transformation
6. Release notes processing
7. Array transformation
8. Dictionary lookup
9. Top N filtering
10. String aggregation
### ERROR_PATTERNS.md (730 lines)
**Top 5 errors with solutions**
1. ModuleNotFoundError (external libraries)
2. Empty code / missing return
3. KeyError (dictionary access)
4. IndexError (list access)
5. Incorrect return format
- Error prevention checklist
- Quick fix reference
- Testing patterns
---
## Integration with Other Skills
This skill works with:
### n8n Expression Syntax
- Python uses code syntax, not {{}} expressions
- Data access patterns differ ($ vs _)
### n8n MCP Tools Expert
- Use MCP tools to validate Code node configurations
- Check node setup with `get_node_essentials`
### n8n Workflow Patterns
- Code nodes fit into larger workflow patterns
- Often used after HTTP Request or Webhook nodes
### n8n Code JavaScript
- Compare Python vs JavaScript approaches
- Understand when to use which language
- JavaScript recommended for 95% of cases
### n8n Node Configuration
- Configure Code node mode (All Items vs Each Item)
- Set up proper connections
---
## Success Metrics
After using this skill, you should be able to:
- [ ] **Know the limitation**: Python has NO external libraries
- [ ] **Choose language**: JavaScript for 95% of cases, Python when needed
- [ ] **Access data**: Use `_input.all()`, `_input.first()`, `_input.item`
- [ ] **Handle webhooks**: Access data via `_json["body"]`
- [ ] **Return properly**: Always return `[{"json": {...}}]`
- [ ] **Avoid KeyError**: Use `.get()` for dictionary access
- [ ] **Use standard library**: Know what's available (json, datetime, re, etc.)
- [ ] **Prevent errors**: Avoid top 5 common errors
- [ ] **Choose alternatives**: Use n8n nodes when libraries needed
- [ ] **Write production code**: Use proven patterns
---
## Quick Reference
### Data Access
```python
all_items = _input.all()
first_item = _input.first()
current_item = _input.item # Each Item mode only
other_node = _node["NodeName"]
```
### Webhook Data
```python
webhook = _input.first()["json"]
body = webhook.get("body", {})
name = body.get("name")
```
### Safe Dictionary Access
```python
# ✅ Use .get() with defaults
value = data.get("field", "default")
# ❌ Risky - may raise KeyError
value = data["field"]
```
### Return Format
```python
# ✅ Correct format
return [{"json": {"result": "success"}}]
# ❌ Wrong - plain dict
return {"result": "success"}
```
### Standard Library
```python
# ✅ Available
import json
import datetime
import re
import base64
import hashlib
# ❌ NOT available
import requests # ModuleNotFoundError!
import pandas # ModuleNotFoundError!
import numpy # ModuleNotFoundError!
```
---
## Common Use Cases
### Use Case 1: Process Webhook Data
```python
webhook = _input.first()["json"]
body = webhook.get("body", {})
return [{
"json": {
"name": body.get("name"),
"email": body.get("email"),
"processed": True
}
}]
```
### Use Case 2: Filter and Transform
```python
all_items = _input.all()
active = [
{"json": {**item["json"], "filtered": True}}
for item in all_items
if item["json"].get("status") == "active"
]
return active
```
### Use Case 3: Aggregate Statistics
```python
import statistics
all_items = _input.all()
amounts = [item["json"].get("amount", 0) for item in all_items]
return [{
"json": {
"total": sum(amounts),
"average": statistics.mean(amounts) if amounts else 0,
"count": len(amounts)
}
}]
```
### Use Case 4: Parse JSON String
```python
import json
data = _input.first()["json"]["body"]
json_string = data.get("payload", "{}")
try:
parsed = json.loads(json_string)
return [{"json": parsed}]
except json.JSONDecodeError:
return [{"json": {"error": "Invalid JSON"}}]
```
---
## Limitations and Workarounds
### Limitation 1: No HTTP Requests Library
**Problem**: No `requests` library
**Workaround**: Use HTTP Request node or JavaScript
### Limitation 2: No Data Analysis Library
**Problem**: No `pandas` or `numpy`
**Workaround**: Use list comprehensions and standard library
### Limitation 3: No Database Drivers
**Problem**: No `psycopg2`, `pymongo`, etc.
**Workaround**: Use n8n database nodes (Postgres, MySQL, MongoDB)
### Limitation 4: No Web Scraping
**Problem**: No `beautifulsoup4` or `selenium`
**Workaround**: Use HTML Extract node
---
## Best Practices
1. **Use JavaScript for most cases** (95% recommendation)
2. **Use .get() for dictionaries** (avoid KeyError)
3. **Check lengths before indexing** (avoid IndexError)
4. **Always return proper format**: `[{"json": {...}}]`
5. **Access webhook data via ["body"]**
6. **Use standard library only** (no external imports)
7. **Handle empty input** (check `if items:`)
8. **Test both modes** (All Items and Each Item)
---
## When Python is the Right Choice
Use Python when:
- Complex text processing (re module)
- Mathematical calculations (math, statistics)
- Date/time manipulation (datetime)
- Cryptographic operations (hashlib)
- You have existing Python logic to reuse
- Team is more comfortable with Python
Use JavaScript instead when:
- Making HTTP requests
- Working with dates (Luxon included)
- Most data transformations
- When in doubt
---
## Learning Path
**Beginner**:
1. Read SKILL.md - Understand the limitation
2. Try DATA_ACCESS.md examples - Learn `_input` patterns
3. Practice safe dictionary access with `.get()`
**Intermediate**:
4. Study STANDARD_LIBRARY.md - Know what's available
5. Try COMMON_PATTERNS.md examples - Use proven patterns
6. Learn ERROR_PATTERNS.md - Avoid common mistakes
**Advanced**:
7. Combine multiple patterns
8. Use standard library effectively
9. Know when to switch to JavaScript
10. Write production-ready code
---
## Support
**Questions?**
- Check ERROR_PATTERNS.md for common issues
- Review COMMON_PATTERNS.md for examples
- Consider using JavaScript instead
**Related Skills**:
- n8n Code JavaScript - Alternative (recommended for 95% of cases)
- n8n Expression Syntax - For {{}} expressions in other nodes
- n8n Workflow Patterns - Bigger picture workflow design
---
## Version
**Version**: 1.0.0
**Status**: Production Ready
**Compatibility**: n8n Code node (Python mode)
---
## Credits
Part of the n8n-skills project.
**Conceived by Romuald Członkowski**
- Website: [www.aiadvisors.pl/en](https://www.aiadvisors.pl/en)
- Part of [n8n-mcp project](https://github.com/czlonkowski/n8n-mcp)
---
**Remember**: JavaScript is recommended for 95% of use cases. Use Python only when you specifically need Python's standard library features.

748
.opencode/skills-off/n8n-code-python/SKILL.md Normal file
View File

@@ -0,0 +1,748 @@
---
name: n8n-code-python
description: Write Python code in n8n Code nodes. Use when writing Python in n8n, using _input/_json/_node syntax, working with standard library, or need to understand Python limitations in n8n Code nodes.
---
# Python Code Node (Beta)
Expert guidance for writing Python code in n8n Code nodes.
---
## ⚠️ Important: JavaScript First
**Recommendation**: Use **JavaScript for 95% of use cases**. Only use Python when:
- You need specific Python standard library functions
- You're significantly more comfortable with Python syntax
- You're doing data transformations better suited to Python
**Why JavaScript is preferred:**
- Full n8n helper functions ($helpers.httpRequest, etc.)
- Luxon DateTime library for advanced date/time operations
- No external library limitations
- Better n8n documentation and community support
---
## Quick Start
```python
# Basic template for Python Code nodes
items = _input.all()
# Process data
processed = []
for item in items:
processed.append({
"json": {
**item["json"],
"processed": True,
"timestamp": datetime.now().isoformat()
}
})
return processed
```
### Essential Rules
1. **Consider JavaScript first** - Use Python only when necessary
2. **Access data**: `_input.all()`, `_input.first()`, or `_input.item`
3. **CRITICAL**: Must return `[{"json": {...}}]` format
4. **CRITICAL**: Webhook data is under `_json["body"]` (not `_json` directly)
5. **CRITICAL LIMITATION**: **No external libraries** (no requests, pandas, numpy)
6. **Standard library only**: json, datetime, re, base64, hashlib, urllib.parse, math, random, statistics
---
## Mode Selection Guide
Same as JavaScript - choose based on your use case:
### Run Once for All Items (Recommended - Default)
**Use this mode for:** 95% of use cases
- **How it works**: Code executes **once** regardless of input count
- **Data access**: `_input.all()` or `_items` array (Native mode)
- **Best for**: Aggregation, filtering, batch processing, transformations
- **Performance**: Faster for multiple items (single execution)
```python
# Example: Calculate total from all items
all_items = _input.all()
total = sum(item["json"].get("amount", 0) for item in all_items)
return [{
"json": {
"total": total,
"count": len(all_items),
"average": total / len(all_items) if all_items else 0
}
}]
```
### Run Once for Each Item
**Use this mode for:** Specialized cases only
- **How it works**: Code executes **separately** for each input item
- **Data access**: `_input.item` or `_item` (Native mode)
- **Best for**: Item-specific logic, independent operations, per-item validation
- **Performance**: Slower for large datasets (multiple executions)
```python
# Example: Add processing timestamp to each item
item = _input.item
return [{
"json": {
**item["json"],
"processed": True,
"processed_at": datetime.now().isoformat()
}
}]
```
---
## Python Modes: Beta vs Native
n8n offers two Python execution modes:
### Python (Beta) - Recommended
- **Use**: `_input`, `_json`, `_node` helper syntax
- **Best for**: Most Python use cases
- **Helpers available**: `_now`, `_today`, `_jmespath()`
- **Import**: `from datetime import datetime`
```python
# Python (Beta) example
items = _input.all()
now = _now # Built-in datetime object
return [{
"json": {
"count": len(items),
"timestamp": now.isoformat()
}
}]
```
### Python (Native) (Beta)
- **Use**: `_items`, `_item` variables only
- **No helpers**: No `_input`, `_now`, etc.
- **More limited**: Standard Python only
- **Use when**: Need pure Python without n8n helpers
```python
# Python (Native) example
processed = []
for item in _items:
processed.append({
"json": {
"id": item["json"].get("id"),
"processed": True
}
})
return processed
```
**Recommendation**: Use **Python (Beta)** for better n8n integration.
---
## Data Access Patterns
### Pattern 1: _input.all() - Most Common
**Use when**: Processing arrays, batch operations, aggregations
```python
# Get all items from previous node
all_items = _input.all()
# Filter, transform as needed
valid = [item for item in all_items if item["json"].get("status") == "active"]
processed = []
for item in valid:
processed.append({
"json": {
"id": item["json"]["id"],
"name": item["json"]["name"]
}
})
return processed
```
### Pattern 2: _input.first() - Very Common
**Use when**: Working with single objects, API responses
```python
# Get first item only
first_item = _input.first()
data = first_item["json"]
return [{
"json": {
"result": process_data(data),
"processed_at": datetime.now().isoformat()
}
}]
```
### Pattern 3: _input.item - Each Item Mode Only
**Use when**: In "Run Once for Each Item" mode
```python
# Current item in loop (Each Item mode only)
current_item = _input.item
return [{
"json": {
**current_item["json"],
"item_processed": True
}
}]
```
### Pattern 4: _node - Reference Other Nodes
**Use when**: Need data from specific nodes in workflow
```python
# Get output from specific node
webhook_data = _node["Webhook"]["json"]
http_data = _node["HTTP Request"]["json"]
return [{
"json": {
"combined": {
"webhook": webhook_data,
"api": http_data
}
}
}]
```
**See**: [DATA_ACCESS.md](DATA_ACCESS.md) for comprehensive guide
---
## Critical: Webhook Data Structure
**MOST COMMON MISTAKE**: Webhook data is nested under `["body"]`
```python
# ❌ WRONG - Will raise KeyError
name = _json["name"]
email = _json["email"]
# ✅ CORRECT - Webhook data is under ["body"]
name = _json["body"]["name"]
email = _json["body"]["email"]
# ✅ SAFER - Use .get() for safe access
webhook_data = _json.get("body", {})
name = webhook_data.get("name")
```
**Why**: Webhook node wraps all request data under `body` property. This includes POST data, query parameters, and JSON payloads.
**See**: [DATA_ACCESS.md](DATA_ACCESS.md) for full webhook structure details
---
## Return Format Requirements
**CRITICAL RULE**: Always return list of dictionaries with `"json"` key
### Correct Return Formats
```python
# ✅ Single result
return [{
"json": {
"field1": value1,
"field2": value2
}
}]
# ✅ Multiple results
return [
{"json": {"id": 1, "data": "first"}},
{"json": {"id": 2, "data": "second"}}
]
# ✅ List comprehension
transformed = [
{"json": {"id": item["json"]["id"], "processed": True}}
for item in _input.all()
if item["json"].get("valid")
]
return transformed
# ✅ Empty result (when no data to return)
return []
# ✅ Conditional return
if should_process:
return [{"json": processed_data}]
else:
return []
```
### Incorrect Return Formats
```python
# ❌ WRONG: Dictionary without list wrapper
return {
"json": {"field": value}
}
# ❌ WRONG: List without json wrapper
return [{"field": value}]
# ❌ WRONG: Plain string
return "processed"
# ❌ WRONG: Incomplete structure
return [{"data": value}] # Should be {"json": value}
```
**Why it matters**: Next nodes expect list format. Incorrect format causes workflow execution to fail.
**See**: [ERROR_PATTERNS.md](ERROR_PATTERNS.md) #2 for detailed error solutions
---
## Critical Limitation: No External Libraries
**MOST IMPORTANT PYTHON LIMITATION**: Cannot import external packages
### What's NOT Available
```python
# ❌ NOT AVAILABLE - Will raise ModuleNotFoundError
import requests # ❌ No
import pandas # ❌ No
import numpy # ❌ No
import scipy # ❌ No
from bs4 import BeautifulSoup # ❌ No
import lxml # ❌ No
```
### What IS Available (Standard Library)
```python
# ✅ AVAILABLE - Standard library only
import json # ✅ JSON parsing
import datetime # ✅ Date/time operations
import re # ✅ Regular expressions
import base64 # ✅ Base64 encoding/decoding
import hashlib # ✅ Hashing functions
import urllib.parse # ✅ URL parsing
import math # ✅ Math functions
import random # ✅ Random numbers
import statistics # ✅ Statistical functions
```
### Workarounds
**Need HTTP requests?**
- ✅ Use **HTTP Request node** before Code node
- ✅ Or switch to **JavaScript** and use `$helpers.httpRequest()`
**Need data analysis (pandas/numpy)?**
- ✅ Use Python **statistics** module for basic stats
- ✅ Or switch to **JavaScript** for most operations
- ✅ Manual calculations with lists and dictionaries
**Need web scraping (BeautifulSoup)?**
- ✅ Use **HTTP Request node** + **HTML Extract node**
- ✅ Or switch to **JavaScript** with regex/string methods
**See**: [STANDARD_LIBRARY.md](STANDARD_LIBRARY.md) for complete reference
---
## Common Patterns Overview
Based on production workflows, here are the most useful Python patterns:
### 1. Data Transformation
Transform all items with list comprehensions
```python
items = _input.all()
return [
{
"json": {
"id": item["json"].get("id"),
"name": item["json"].get("name", "Unknown").upper(),
"processed": True
}
}
for item in items
]
```
### 2. Filtering & Aggregation
Sum, filter, count with built-in functions
```python
items = _input.all()
total = sum(item["json"].get("amount", 0) for item in items)
valid_items = [item for item in items if item["json"].get("amount", 0) > 0]
return [{
"json": {
"total": total,
"count": len(valid_items)
}
}]
```
### 3. String Processing with Regex
Extract patterns from text
```python
import re
items = _input.all()
email_pattern = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
all_emails = []
for item in items:
text = item["json"].get("text", "")
emails = re.findall(email_pattern, text)
all_emails.extend(emails)
# Remove duplicates
unique_emails = list(set(all_emails))
return [{
"json": {
"emails": unique_emails,
"count": len(unique_emails)
}
}]
```
### 4. Data Validation
Validate and clean data
```python
items = _input.all()
validated = []
for item in items:
data = item["json"]
errors = []
# Validate fields
if not data.get("email"):
errors.append("Email required")
if not data.get("name"):
errors.append("Name required")
validated.append({
"json": {
**data,
"valid": len(errors) == 0,
"errors": errors if errors else None
}
})
return validated
```
### 5. Statistical Analysis
Calculate statistics with statistics module
```python
from statistics import mean, median, stdev
items = _input.all()
values = [item["json"].get("value", 0) for item in items if "value" in item["json"]]
if values:
return [{
"json": {
"mean": mean(values),
"median": median(values),
"stdev": stdev(values) if len(values) > 1 else 0,
"min": min(values),
"max": max(values),
"count": len(values)
}
}]
else:
return [{"json": {"error": "No values found"}}]
```
**See**: [COMMON_PATTERNS.md](COMMON_PATTERNS.md) for 10 detailed Python patterns
---
## Error Prevention - Top 5 Mistakes
### #1: Importing External Libraries (Python-Specific!)
```python
# ❌ WRONG: Trying to import external library
import requests # ModuleNotFoundError!
# ✅ CORRECT: Use HTTP Request node or JavaScript
# Add HTTP Request node before Code node
# OR switch to JavaScript and use $helpers.httpRequest()
```
### #2: Empty Code or Missing Return
```python
# ❌ WRONG: No return statement
items = _input.all()
# Processing...
# Forgot to return!
# ✅ CORRECT: Always return data
items = _input.all()
# Processing...
return [{"json": item["json"]} for item in items]
```
### #3: Incorrect Return Format
```python
# ❌ WRONG: Returning dict instead of list
return {"json": {"result": "success"}}
# ✅ CORRECT: List wrapper required
return [{"json": {"result": "success"}}]
```
### #4: KeyError on Dictionary Access
```python
# ❌ WRONG: Direct access crashes if missing
name = _json["user"]["name"] # KeyError!
# ✅ CORRECT: Use .get() for safe access
name = _json.get("user", {}).get("name", "Unknown")
```
### #5: Webhook Body Nesting
```python
# ❌ WRONG: Direct access to webhook data
email = _json["email"] # KeyError!
# ✅ CORRECT: Webhook data under ["body"]
email = _json["body"]["email"]
# ✅ BETTER: Safe access with .get()
email = _json.get("body", {}).get("email", "no-email")
```
**See**: [ERROR_PATTERNS.md](ERROR_PATTERNS.md) for comprehensive error guide
---
## Standard Library Reference
### Most Useful Modules
```python
# JSON operations
import json
data = json.loads(json_string)
json_output = json.dumps({"key": "value"})
# Date/time
from datetime import datetime, timedelta
now = datetime.now()
tomorrow = now + timedelta(days=1)
formatted = now.strftime("%Y-%m-%d")
# Regular expressions
import re
matches = re.findall(r'\d+', text)
cleaned = re.sub(r'[^\w\s]', '', text)
# Base64 encoding
import base64
encoded = base64.b64encode(data).decode()
decoded = base64.b64decode(encoded)
# Hashing
import hashlib
hash_value = hashlib.sha256(text.encode()).hexdigest()
# URL parsing
import urllib.parse
params = urllib.parse.urlencode({"key": "value"})
parsed = urllib.parse.urlparse(url)
# Statistics
from statistics import mean, median, stdev
average = mean([1, 2, 3, 4, 5])
```
**See**: [STANDARD_LIBRARY.md](STANDARD_LIBRARY.md) for complete reference
---
## Best Practices
### 1. Always Use .get() for Dictionary Access
```python
# ✅ SAFE: Won't crash if field missing
value = item["json"].get("field", "default")
# ❌ RISKY: Crashes if field doesn't exist
value = item["json"]["field"]
```
### 2. Handle None/Null Values Explicitly
```python
# ✅ GOOD: Default to 0 if None
amount = item["json"].get("amount") or 0
# ✅ GOOD: Check for None explicitly
text = item["json"].get("text")
if text is None:
text = ""
```
### 3. Use List Comprehensions for Filtering
```python
# ✅ PYTHONIC: List comprehension
valid = [item for item in items if item["json"].get("active")]
# ❌ VERBOSE: Manual loop
valid = []
for item in items:
if item["json"].get("active"):
valid.append(item)
```
### 4. Return Consistent Structure
```python
# ✅ CONSISTENT: Always list with "json" key
return [{"json": result}] # Single result
return results # Multiple results (already formatted)
return [] # No results
```
### 5. Debug with print() Statements
```python
# Debug statements appear in browser console (F12)
items = _input.all()
print(f"Processing {len(items)} items")
print(f"First item: {items[0] if items else 'None'}")
```
---
## When to Use Python vs JavaScript
### Use Python When:
- ✅ You need `statistics` module for statistical operations
- ✅ You're significantly more comfortable with Python syntax
- ✅ Your logic maps well to list comprehensions
- ✅ You need specific standard library functions
### Use JavaScript When:
- ✅ You need HTTP requests ($helpers.httpRequest())
- ✅ You need advanced date/time (DateTime/Luxon)
- ✅ You want better n8n integration
- ✅ **For 95% of use cases** (recommended)
### Consider Other Nodes When:
- ❌ Simple field mapping → Use **Set** node
- ❌ Basic filtering → Use **Filter** node
- ❌ Simple conditionals → Use **IF** or **Switch** node
- ❌ HTTP requests only → Use **HTTP Request** node
---
## Integration with Other Skills
### Works With:
**n8n Expression Syntax**:
- Expressions use `{{ }}` syntax in other nodes
- Code nodes use Python directly (no `{{ }}`)
- When to use expressions vs code
**n8n MCP Tools Expert**:
- How to find Code node: `search_nodes({query: "code"})`
- Get configuration help: `get_node_essentials("nodes-base.code")`
- Validate code: `validate_node_operation()`
**n8n Node Configuration**:
- Mode selection (All Items vs Each Item)
- Language selection (Python vs JavaScript)
- Understanding property dependencies
**n8n Workflow Patterns**:
- Code nodes in transformation step
- When to use Python vs JavaScript in patterns
**n8n Validation Expert**:
- Validate Code node configuration
- Handle validation errors
- Auto-fix common issues
**n8n Code JavaScript**:
- When to use JavaScript instead
- Comparison of JavaScript vs Python features
- Migration from Python to JavaScript
---
## Quick Reference Checklist
Before deploying Python Code nodes, verify:
- [ ] **Considered JavaScript first** - Using Python only when necessary
- [ ] **Code is not empty** - Must have meaningful logic
- [ ] **Return statement exists** - Must return list of dictionaries
- [ ] **Proper return format** - Each item: `{"json": {...}}`
- [ ] **Data access correct** - Using `_input.all()`, `_input.first()`, or `_input.item`
- [ ] **No external imports** - Only standard library (json, datetime, re, etc.)
- [ ] **Safe dictionary access** - Using `.get()` to avoid KeyError
- [ ] **Webhook data** - Access via `["body"]` if from webhook
- [ ] **Mode selection** - "All Items" for most cases
- [ ] **Output consistent** - All code paths return same structure
---
## Additional Resources
### Related Files
- [DATA_ACCESS.md](DATA_ACCESS.md) - Comprehensive Python data access patterns
- [COMMON_PATTERNS.md](COMMON_PATTERNS.md) - 10 Python patterns for n8n
- [ERROR_PATTERNS.md](ERROR_PATTERNS.md) - Top 5 errors and solutions
- [STANDARD_LIBRARY.md](STANDARD_LIBRARY.md) - Complete standard library reference
### n8n Documentation
- Code Node Guide: https://docs.n8n.io/code/code-node/
- Python in n8n: https://docs.n8n.io/code/builtin/python-modules/
---
**Ready to write Python in n8n Code nodes - but consider JavaScript first!** Use Python for specific needs, reference the error patterns guide to avoid common mistakes, and leverage the standard library effectively.

974
.opencode/skills-off/n8n-code-python/STANDARD_LIBRARY.md Normal file
View File

@@ -0,0 +1,974 @@
# Standard Library Reference - Python Code Node
Complete guide to available Python standard library modules in n8n Code nodes.
---
## ⚠️ Critical Limitation
**NO EXTERNAL LIBRARIES AVAILABLE**
Python Code nodes in n8n have **ONLY** the Python standard library. No pip packages.
```python
# ❌ NOT AVAILABLE - Will cause ModuleNotFoundError
import requests # No HTTP library!
import pandas # No data analysis!
import numpy # No numerical computing!
import bs4 # No web scraping!
import selenium # No browser automation!
import psycopg2 # No database drivers!
import pymongo # No MongoDB!
import sqlalchemy # No ORMs!
# ✅ AVAILABLE - Standard library only
import json
import datetime
import re
import base64
import hashlib
import urllib.parse
import urllib.request
import math
import random
import statistics
```
**Recommendation**: Use **JavaScript** for 95% of use cases. JavaScript has more capabilities in n8n.
---
## Available Modules
### Priority 1: Most Useful (Use These)
1. **json** - JSON parsing and generation
2. **datetime** - Date and time operations
3. **re** - Regular expressions
4. **base64** - Base64 encoding/decoding
5. **hashlib** - Hashing (MD5, SHA256, etc.)
6. **urllib.parse** - URL parsing and encoding
### Priority 2: Moderately Useful
7. **math** - Mathematical functions
8. **random** - Random number generation
9. **statistics** - Statistical functions
10. **collections** - Specialized data structures
### Priority 3: Occasionally Useful
11. **itertools** - Iterator tools
12. **functools** - Higher-order functions
13. **operator** - Standard operators as functions
14. **string** - String constants and templates
15. **textwrap** - Text wrapping utilities
---
## Module 1: json - JSON Operations
**Most common module** - Parse and generate JSON data.
### Parse JSON String
```python
import json
# Parse JSON string to Python dict
json_string = '{"name": "Alice", "age": 30}'
data = json.loads(json_string)
return [{
"json": {
"name": data["name"],
"age": data["age"],
"parsed": True
}
}]
```
### Generate JSON String
```python
import json
# Convert Python dict to JSON string
data = {
"users": [
{"id": 1, "name": "Alice"},
{"id": 2, "name": "Bob"}
],
"total": 2
}
json_string = json.dumps(data, indent=2)
return [{
"json": {
"json_output": json_string,
"length": len(json_string)
}
}]
```
### Handle JSON Errors
```python
import json
webhook_data = _input.first()["json"]["body"]
json_string = webhook_data.get("data", "")
try:
parsed = json.loads(json_string)
status = "valid"
error = None
except json.JSONDecodeError as e:
parsed = None
status = "invalid"
error = str(e)
return [{
"json": {
"status": status,
"data": parsed,
"error": error
}
}]
```
### Pretty Print JSON
```python
import json
# Format JSON with indentation
data = _input.first()["json"]
pretty_json = json.dumps(data, indent=2, sort_keys=True)
return [{
"json": {
"formatted": pretty_json
}
}]
```
---
## Module 2: datetime - Date and Time
**Very common** - Date parsing, formatting, calculations.
### Current Date and Time
```python
from datetime import datetime
now = datetime.now()
return [{
"json": {
"timestamp": now.isoformat(),
"date": now.strftime("%Y-%m-%d"),
"time": now.strftime("%H:%M:%S"),
"formatted": now.strftime("%B %d, %Y at %I:%M %p")
}
}]
```
### Parse Date String
```python
from datetime import datetime
date_string = "2025-01-15T14:30:00"
dt = datetime.fromisoformat(date_string)
return [{
"json": {
"year": dt.year,
"month": dt.month,
"day": dt.day,
"hour": dt.hour,
"weekday": dt.strftime("%A")
}
}]
```
### Date Calculations
```python
from datetime import datetime, timedelta
now = datetime.now()
# Calculate future/past dates
tomorrow = now + timedelta(days=1)
yesterday = now - timedelta(days=1)
next_week = now + timedelta(weeks=1)
one_hour_ago = now - timedelta(hours=1)
return [{
"json": {
"now": now.isoformat(),
"tomorrow": tomorrow.isoformat(),
"yesterday": yesterday.isoformat(),
"next_week": next_week.isoformat(),
"one_hour_ago": one_hour_ago.isoformat()
}
}]
```
### Compare Dates
```python
from datetime import datetime
date1 = datetime(2025, 1, 15)
date2 = datetime(2025, 1, 20)
# Calculate difference
diff = date2 - date1
return [{
"json": {
"days_difference": diff.days,
"seconds_difference": diff.total_seconds(),
"date1_is_earlier": date1 < date2,
"date2_is_later": date2 > date1
}
}]
```
### Format Dates
```python
from datetime import datetime
dt = datetime.now()
return [{
"json": {
"iso": dt.isoformat(),
"us_format": dt.strftime("%m/%d/%Y"),
"eu_format": dt.strftime("%d/%m/%Y"),
"long_format": dt.strftime("%A, %B %d, %Y"),
"time_12h": dt.strftime("%I:%M %p"),
"time_24h": dt.strftime("%H:%M:%S")
}
}]
```
---
## Module 3: re - Regular Expressions
**Common** - Pattern matching, text extraction, validation.
### Pattern Matching
```python
import re
text = "Email: alice@example.com, Phone: 555-1234"
# Find email
email_match = re.search(r'\b[\w.-]+@[\w.-]+\.\w+\b', text)
email = email_match.group(0) if email_match else None
# Find phone
phone_match = re.search(r'\d{3}-\d{4}', text)
phone = phone_match.group(0) if phone_match else None
return [{
"json": {
"email": email,
"phone": phone
}
}]
```
### Extract All Matches
```python
import re
text = "Tags: #python #automation #workflow #n8n"
# Find all hashtags
hashtags = re.findall(r'#(\w+)', text)
return [{
"json": {
"tags": hashtags,
"count": len(hashtags)
}
}]
```
### Replace Patterns
```python
import re
text = "Price: $99.99, Discount: $10.00"
# Remove dollar signs
cleaned = re.sub(r'\$', '', text)
# Replace multiple spaces with single space
normalized = re.sub(r'\s+', ' ', cleaned)
return [{
"json": {
"original": text,
"cleaned": cleaned,
"normalized": normalized
}
}]
```
### Validate Format
```python
import re
email = _input.first()["json"]["body"].get("email", "")
# Email validation pattern
email_pattern = r'^[\w.-]+@[\w.-]+\.\w+$'
is_valid = bool(re.match(email_pattern, email))
return [{
"json": {
"email": email,
"valid": is_valid
}
}]
```
### Split on Pattern
```python
import re
text = "apple,banana;orange|grape"
# Split on multiple delimiters
items = re.split(r'[,;|]', text)
# Clean up whitespace
items = [item.strip() for item in items]
return [{
"json": {
"items": items,
"count": len(items)
}
}]
```
---
## Module 4: base64 - Encoding/Decoding
**Common** - Encode binary data, API authentication.
### Encode String to Base64
```python
import base64
text = "Hello, World!"
# Encode to base64
encoded_bytes = base64.b64encode(text.encode('utf-8'))
encoded_string = encoded_bytes.decode('utf-8')
return [{
"json": {
"original": text,
"encoded": encoded_string
}
}]
```
### Decode Base64 to String
```python
import base64
encoded = "SGVsbG8sIFdvcmxkIQ=="
# Decode from base64
decoded_bytes = base64.b64decode(encoded)
decoded_string = decoded_bytes.decode('utf-8')
return [{
"json": {
"encoded": encoded,
"decoded": decoded_string
}
}]
```
### Basic Auth Header
```python
import base64
username = "admin"
password = "secret123"
# Create Basic Auth header
credentials = f"{username}:{password}"
encoded = base64.b64encode(credentials.encode('utf-8')).decode('utf-8')
auth_header = f"Basic {encoded}"
return [{
"json": {
"authorization": auth_header
}
}]
```
---
## Module 5: hashlib - Hashing
**Common** - Generate checksums, hash passwords, create IDs.
### MD5 Hash
```python
import hashlib
text = "Hello, World!"
# Generate MD5 hash
md5_hash = hashlib.md5(text.encode('utf-8')).hexdigest()
return [{
"json": {
"original": text,
"md5": md5_hash
}
}]
```
### SHA256 Hash
```python
import hashlib
data = _input.first()["json"]["body"]
text = data.get("password", "")
# Generate SHA256 hash (more secure than MD5)
sha256_hash = hashlib.sha256(text.encode('utf-8')).hexdigest()
return [{
"json": {
"hashed": sha256_hash
}
}]
```
### Generate Unique ID
```python
import hashlib
from datetime import datetime
# Create unique ID from multiple values
unique_string = f"{datetime.now().isoformat()}-{_json.get('user_id', 'unknown')}"
unique_id = hashlib.sha256(unique_string.encode('utf-8')).hexdigest()[:16]
return [{
"json": {
"id": unique_id,
"generated_at": datetime.now().isoformat()
}
}]
```
---
## Module 6: urllib.parse - URL Operations
**Common** - Parse URLs, encode parameters.
### Parse URL
```python
from urllib.parse import urlparse
url = "https://example.com/path?key=value&foo=bar#section"
parsed = urlparse(url)
return [{
"json": {
"scheme": parsed.scheme, # "https"
"netloc": parsed.netloc, # "example.com"
"path": parsed.path, # "/path"
"query": parsed.query, # "key=value&foo=bar"
"fragment": parsed.fragment # "section"
}
}]
```
### URL Encode Parameters
```python
from urllib.parse import urlencode
params = {
"name": "Alice Smith",
"email": "alice@example.com",
"message": "Hello, World!"
}
# Encode parameters for URL
encoded = urlencode(params)
return [{
"json": {
"query_string": encoded,
"full_url": f"https://api.example.com/submit?{encoded}"
}
}]
```
### Parse Query String
```python
from urllib.parse import parse_qs
query_string = "name=Alice&age=30&tags=python&tags=n8n"
# Parse query string
params = parse_qs(query_string)
return [{
"json": {
"name": params.get("name", [""])[0],
"age": int(params.get("age", ["0"])[0]),
"tags": params.get("tags", [])
}
}]
```
### URL Encode/Decode Strings
```python
from urllib.parse import quote, unquote
text = "Hello, World! 你好"
# URL encode
encoded = quote(text)
# URL decode
decoded = unquote(encoded)
return [{
"json": {
"original": text,
"encoded": encoded,
"decoded": decoded
}
}]
```
---
## Module 7: math - Mathematical Operations
**Moderately useful** - Advanced math functions.
### Basic Math Functions
```python
import math
number = 16.7
return [{
"json": {
"ceiling": math.ceil(number), # 17
"floor": math.floor(number), # 16
"rounded": round(number), # 17
"square_root": math.sqrt(16), # 4.0
"power": math.pow(2, 3), # 8.0
"absolute": math.fabs(-5.5) # 5.5
}
}]
```
### Trigonometry
```python
import math
angle_degrees = 45
angle_radians = math.radians(angle_degrees)
return [{
"json": {
"sine": math.sin(angle_radians),
"cosine": math.cos(angle_radians),
"tangent": math.tan(angle_radians),
"pi": math.pi,
"e": math.e
}
}]
```
### Logarithms
```python
import math
number = 100
return [{
"json": {
"log10": math.log10(number), # 2.0
"natural_log": math.log(number), # 4.605...
"log2": math.log2(number) # 6.644...
}
}]
```
---
## Module 8: random - Random Numbers
**Moderately useful** - Generate random data, sampling.
### Random Numbers
```python
import random
return [{
"json": {
"random_float": random.random(), # 0.0 to 1.0
"random_int": random.randint(1, 100), # 1 to 100
"random_range": random.randrange(0, 100, 5) # 0, 5, 10, ..., 95
}
}]
```
### Random Choice
```python
import random
colors = ["red", "green", "blue", "yellow"]
users = [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
return [{
"json": {
"random_color": random.choice(colors),
"random_user": random.choice(users)
}
}]
```
### Shuffle List
```python
import random
items = [1, 2, 3, 4, 5]
shuffled = items.copy()
random.shuffle(shuffled)
return [{
"json": {
"original": items,
"shuffled": shuffled
}
}]
```
### Random Sample
```python
import random
items = list(range(1, 101))
# Get 10 random items without replacement
sample = random.sample(items, 10)
return [{
"json": {
"sample": sample,
"count": len(sample)
}
}]
```
---
## Module 9: statistics - Statistical Functions
**Moderately useful** - Calculate stats from data.
### Basic Statistics
```python
import statistics
numbers = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
return [{
"json": {
"mean": statistics.mean(numbers), # 55.0
"median": statistics.median(numbers), # 55.0
"mode": statistics.mode([1, 2, 2, 3]), # 2
"stdev": statistics.stdev(numbers), # 30.28...
"variance": statistics.variance(numbers) # 916.67...
}
}]
```
### Aggregate from Items
```python
import statistics
all_items = _input.all()
# Extract amounts
amounts = [item["json"].get("amount", 0) for item in all_items]
if amounts:
return [{
"json": {
"count": len(amounts),
"total": sum(amounts),
"average": statistics.mean(amounts),
"median": statistics.median(amounts),
"min": min(amounts),
"max": max(amounts),
"range": max(amounts) - min(amounts)
}
}]
else:
return [{"json": {"error": "No data"}}]
```
---
## Workarounds for Missing Libraries
### HTTP Requests (No requests library)
```python
# ❌ Can't use requests library
# import requests # ModuleNotFoundError!
# ✅ Use HTTP Request node instead
# Add HTTP Request node BEFORE Code node
# Access the response in Code node
response_data = _input.first()["json"]
return [{
"json": {
"status": response_data.get("status"),
"data": response_data.get("body"),
"processed": True
}
}]
```
### Data Processing (No pandas)
```python
# ❌ Can't use pandas
# import pandas as pd # ModuleNotFoundError!
# ✅ Use Python's built-in list comprehensions
all_items = _input.all()
# Filter
active_items = [
item for item in all_items
if item["json"].get("status") == "active"
]
# Group by
from collections import defaultdict
grouped = defaultdict(list)
for item in all_items:
category = item["json"].get("category", "other")
grouped[category].append(item["json"])
# Aggregate
import statistics
amounts = [item["json"].get("amount", 0) for item in all_items]
total = sum(amounts)
average = statistics.mean(amounts) if amounts else 0
return [{
"json": {
"active_count": len(active_items),
"grouped": dict(grouped),
"total": total,
"average": average
}
}]
```
### Database Operations (No drivers)
```python
# ❌ Can't use database drivers
# import psycopg2 # ModuleNotFoundError!
# import pymongo # ModuleNotFoundError!
# ✅ Use n8n database nodes instead
# Add Postgres/MySQL/MongoDB node BEFORE Code node
# Process results in Code node
db_results = _input.first()["json"]
return [{
"json": {
"record_count": len(db_results) if isinstance(db_results, list) else 1,
"processed": True
}
}]
```
---
## Complete Standard Library List
**Available** (commonly useful):
- json
- datetime, time
- re
- base64
- hashlib
- urllib.parse, urllib.request, urllib.error
- math
- random
- statistics
- collections (defaultdict, Counter, namedtuple)
- itertools
- functools
- operator
- string
- textwrap
**Available** (less common):
- os.path (path operations only)
- copy
- typing
- enum
- decimal
- fractions
**NOT Available** (external libraries):
- requests (HTTP)
- pandas (data analysis)
- numpy (numerical computing)
- bs4/beautifulsoup4 (HTML parsing)
- selenium (browser automation)
- psycopg2, pymongo, sqlalchemy (databases)
- flask, fastapi (web frameworks)
- pillow (image processing)
- openpyxl, xlsxwriter (Excel)
---
## Best Practices
### 1. Use Standard Library When Possible
```python
# ✅ GOOD: Use standard library
import json
import datetime
import re
data = _input.first()["json"]
processed = json.loads(data.get("json_string", "{}"))
return [{"json": processed}]
```
### 2. Fall Back to n8n Nodes
```python
# For operations requiring external libraries,
# use n8n nodes instead:
# - HTTP Request for API calls
# - Postgres/MySQL for databases
# - Extract from File for parsing
# Then process results in Code node
result = _input.first()["json"]
return [{"json": {"processed": result}}]
```
### 3. Combine Multiple Modules
```python
import json
import base64
import hashlib
from datetime import datetime
# Combine modules for complex operations
data = _input.first()["json"]["body"]
# Hash sensitive data
user_id = hashlib.sha256(data.get("email", "").encode()).hexdigest()[:16]
# Encode for storage
encoded_data = base64.b64encode(json.dumps(data).encode()).decode()
return [{
"json": {
"user_id": user_id,
"encoded_data": encoded_data,
"timestamp": datetime.now().isoformat()
}
}]
```
---
## Summary
**Most Useful Modules**:
1. json - Parse/generate JSON
2. datetime - Date operations
3. re - Regular expressions
4. base64 - Encoding
5. hashlib - Hashing
6. urllib.parse - URL operations
**Critical Limitation**:
- NO external libraries (requests, pandas, numpy, etc.)
**Recommended Approach**:
- Use **JavaScript** for 95% of use cases
- Use Python only when specifically needed
- Use n8n nodes for operations requiring external libraries
**See Also**:
- [SKILL.md](SKILL.md) - Python Code overview
- [DATA_ACCESS.md](DATA_ACCESS.md) - Data access patterns
- [COMMON_PATTERNS.md](COMMON_PATTERNS.md) - Production patterns
- [ERROR_PATTERNS.md](ERROR_PATTERNS.md) - Avoid common mistakes

251
.opencode/skills-off/n8n-workflow-patterns/README.md Normal file
View File

@@ -0,0 +1,251 @@
# n8n Workflow Patterns
Proven architectural patterns for building n8n workflows.
---
## Purpose
Teaches architectural patterns for building n8n workflows. Provides structure, best practices, and proven approaches for common use cases.
## Activates On
- build workflow
- workflow pattern
- workflow architecture
- workflow structure
- webhook processing
- http api
- api integration
- database sync
- ai agent
- chatbot
- scheduled task
- automation pattern
## File Count
7 files, ~3,700 lines total
## Priority
**HIGH** - Addresses 813 webhook searches (most common use case)
## Dependencies
**n8n-mcp tools**:
- search_nodes (find nodes for patterns)
- get_node (understand node operations)
- search_templates (find example workflows)
- ai_agents_guide (AI pattern guidance)
**Related skills**:
- n8n MCP Tools Expert (find and configure nodes)
- n8n Expression Syntax (write expressions in patterns)
- n8n Node Configuration (configure pattern nodes)
- n8n Validation Expert (validate pattern implementations)
## Coverage
### The 5 Core Patterns
1. **Webhook Processing** (Most Common - 813 searches)
- Receive HTTP requests → Process → Respond
- Critical gotcha: Data under $json.body
- Authentication, validation, error handling
2. **HTTP API Integration** (892 templates)
- Fetch from REST APIs → Transform → Store/Use
- Authentication methods, pagination, rate limiting
- Error handling and retries
3. **Database Operations** (456 templates)
- Read/Write/Sync database data
- Batch processing, transactions, performance
- Security: parameterized queries, read-only access
4. **AI Agent Workflow** (234 templates, 270 AI nodes)
- AI agents with tool access and memory
- 8 AI connection types
- ANY node can be an AI tool
5. **Scheduled Tasks** (28% of all workflows)
- Recurring automation workflows
- Cron schedules, timezone handling
- Monitoring and error handling
### Cross-Cutting Concerns
- Data flow patterns (linear, branching, parallel, loops)
- Error handling strategies
- Performance optimization
- Security best practices
- Testing approaches
- Monitoring and logging
## Evaluations
5 scenarios (100% coverage expected):
1. **eval-001**: Webhook workflow structure
2. **eval-002**: HTTP API integration pattern
3. **eval-003**: Database sync pattern
4. **eval-004**: AI agent workflow with tools
5. **eval-005**: Scheduled report generation
## Key Features
✅ **5 Proven Patterns**: Webhook, HTTP API, Database, AI Agent, Scheduled tasks
✅ **Complete Examples**: Working workflow configurations for each pattern
✅ **Best Practices**: Proven approaches from real-world n8n usage
✅ **Common Gotchas**: Documented mistakes and their fixes
✅ **Integration Guide**: How patterns work with other skills
✅ **Template Examples**: Real examples from 2,653+ n8n templates
## Files
- **SKILL.md** (486 lines) - Pattern overview, selection guide, checklist
- **webhook_processing.md** (554 lines) - Webhook patterns, data structure, auth
- **http_api_integration.md** (763 lines) - REST APIs, pagination, rate limiting
- **database_operations.md** (854 lines) - DB operations, batch processing, security
- **ai_agent_workflow.md** (918 lines) - AI agents, tools, memory, 8 connection types
- **scheduled_tasks.md** (845 lines) - Cron schedules, timezone, monitoring
- **README.md** (this file) - Skill metadata
## Success Metrics
**Expected outcomes**:
- Users select appropriate pattern for their use case
- Workflows follow proven structural patterns
- Common gotchas avoided (webhook $json.body, SQL injection, etc.)
- Proper error handling implemented
- Security best practices followed
## Pattern Selection Stats
Common workflow composition:
**Trigger Distribution**:
- Webhook: 35% (most common)
- Schedule: 28%
- Manual: 22%
- Service triggers: 15%
**Transformation Nodes**:
- Set: 68%
- Code: 42%
- IF: 38%
- Switch: 18%
**Output Channels**:
- HTTP Request: 45%
- Slack: 32%
- Database: 28%
- Email: 24%
**Complexity**:
- Simple (3-5 nodes): 42%
- Medium (6-10 nodes): 38%
- Complex (11+ nodes): 20%
## Critical Insights
**Webhook Processing**:
- 813 searches (most common use case!)
- #1 gotcha: Data under $json.body (not $json directly)
- Must choose response mode: onReceived vs lastNode
**API Integration**:
- Authentication via credentials (never hardcode!)
- Pagination essential for large datasets
- Rate limiting prevents API bans
- continueOnFail: true for error handling
**Database Operations**:
- Always use parameterized queries (SQL injection prevention)
- Batch processing for large datasets
- Read-only access for AI tools
- Transaction handling for multi-step operations
**AI Agents**:
- 8 AI connection types (ai_languageModel, ai_tool, ai_memory, etc.)
- ANY node can be an AI tool (connect to ai_tool port)
- Memory essential for conversations (Window Buffer recommended)
- Tool descriptions critical (AI uses them to decide when to call)
**Scheduled Tasks**:
- Set workflow timezone explicitly (DST handling)
- Prevent overlapping executions (use locks)
- Error Trigger workflow for alerts
- Batch processing for large data
## Workflow Creation Checklist
Every pattern follows this checklist:
### Planning Phase
- [ ] Identify the pattern (webhook, API, database, AI, scheduled)
- [ ] List required nodes (use search_nodes)
- [ ] Understand data flow (input → transform → output)
- [ ] Plan error handling strategy
### Implementation Phase
- [ ] Create workflow with appropriate trigger
- [ ] Add data source nodes
- [ ] Configure authentication/credentials
- [ ] Add transformation nodes (Set, Code, IF)
- [ ] Add output/action nodes
- [ ] Configure error handling
### Validation Phase
- [ ] Validate each node configuration
- [ ] Validate complete workflow
- [ ] Test with sample data
- [ ] Handle edge cases
### Deployment Phase
- [ ] Review workflow settings
- [ ] Activate workflow
- [ ] Monitor first executions
- [ ] Document workflow
## Real Template Examples
**Weather to Slack** (Template #2947):
```
Schedule (daily 8 AM) → HTTP Request (weather) → Set → Slack
```
**Webhook Processing**: 1,085 templates
**HTTP API Integration**: 892 templates
**Database Operations**: 456 templates
**AI Workflows**: 234 templates
Use `search_templates` to find examples for your use case!
## Integration with Other Skills
**Pattern Selection** (this skill):
1. Identify use case
2. Select appropriate pattern
3. Follow pattern structure
**Node Discovery** (n8n MCP Tools Expert):
4. Find nodes for pattern (search_nodes)
5. Understand node operations (get_node)
**Implementation** (n8n Expression Syntax + Node Configuration):
6. Write expressions ({{$json.body.field}})
7. Configure nodes properly
**Validation** (n8n Validation Expert):
8. Validate workflow structure
9. Fix validation errors
## Last Updated
2025-10-20
---
**Part of**: n8n-skills repository
**Conceived by**: Romuald Członkowski - [www.aiadvisors.pl/en](https://www.aiadvisors.pl/en)

411
.opencode/skills-off/n8n-workflow-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,411 @@
---
name: n8n-workflow-patterns
description: Proven workflow architectural patterns from real n8n workflows. Use when building new workflows, designing workflow structure, choosing workflow patterns, planning workflow architecture, or asking about webhook processing, HTTP API integration, database operations, AI agent workflows, or scheduled tasks.
---
# n8n Workflow Patterns
Proven architectural patterns for building n8n workflows.
---
## The 5 Core Patterns
Based on analysis of real workflow usage:
1. **[Webhook Processing](webhook_processing.md)** (Most Common)
- Receive HTTP requests → Process → Output
- Pattern: Webhook → Validate → Transform → Respond/Notify
2. **[HTTP API Integration](http_api_integration.md)**
- Fetch from REST APIs → Transform → Store/Use
- Pattern: Trigger → HTTP Request → Transform → Action → Error Handler
3. **[Database Operations](database_operations.md)**
- Read/Write/Sync database data
- Pattern: Schedule → Query → Transform → Write → Verify
4. **[AI Agent Workflow](ai_agent_workflow.md)**
- AI agents with tools and memory
- Pattern: Trigger → AI Agent (Model + Tools + Memory) → Output
5. **[Scheduled Tasks](scheduled_tasks.md)**
- Recurring automation workflows
- Pattern: Schedule → Fetch → Process → Deliver → Log
---
## Pattern Selection Guide
### When to use each pattern:
**Webhook Processing** - Use when:
- Receiving data from external systems
- Building integrations (Slack commands, form submissions, GitHub webhooks)
- Need instant response to events
- Example: "Receive Stripe payment webhook → Update database → Send confirmation"
**HTTP API Integration** - Use when:
- Fetching data from external APIs
- Synchronizing with third-party services
- Building data pipelines
- Example: "Fetch GitHub issues → Transform → Create Jira tickets"
**Database Operations** - Use when:
- Syncing between databases
- Running database queries on schedule
- ETL workflows
- Example: "Read Postgres records → Transform → Write to MySQL"
**AI Agent Workflow** - Use when:
- Building conversational AI
- Need AI with tool access
- Multi-step reasoning tasks
- Example: "Chat with AI that can search docs, query database, send emails"
**Scheduled Tasks** - Use when:
- Recurring reports or summaries
- Periodic data fetching
- Maintenance tasks
- Example: "Daily: Fetch analytics → Generate report → Email team"
---
## Common Workflow Components
All patterns share these building blocks:
### 1. Triggers
- **Webhook** - HTTP endpoint (instant)
- **Schedule** - Cron-based timing (periodic)
- **Manual** - Click to execute (testing)
- **Polling** - Check for changes (intervals)
### 2. Data Sources
- **HTTP Request** - REST APIs
- **Database nodes** - Postgres, MySQL, MongoDB
- **Service nodes** - Slack, Google Sheets, etc.
- **Code** - Custom JavaScript/Python
### 3. Transformation
- **Set** - Map/transform fields
- **Code** - Complex logic
- **IF/Switch** - Conditional routing
- **Merge** - Combine data streams
### 4. Outputs
- **HTTP Request** - Call APIs
- **Database** - Write data
- **Communication** - Email, Slack, Discord
- **Storage** - Files, cloud storage
### 5. Error Handling
- **Error Trigger** - Catch workflow errors
- **IF** - Check for error conditions
- **Stop and Error** - Explicit failure
- **Continue On Fail** - Per-node setting
---
## Workflow Creation Checklist
When building ANY workflow, follow this checklist:
### Planning Phase
- [ ] Identify the pattern (webhook, API, database, AI, scheduled)
- [ ] List required nodes (use search_nodes)
- [ ] Understand data flow (input → transform → output)
- [ ] Plan error handling strategy
### Implementation Phase
- [ ] Create workflow with appropriate trigger
- [ ] Add data source nodes
- [ ] Configure authentication/credentials
- [ ] Add transformation nodes (Set, Code, IF)
- [ ] Add output/action nodes
- [ ] Configure error handling
### Validation Phase
- [ ] Validate each node configuration (validate_node)
- [ ] Validate complete workflow (validate_workflow)
- [ ] Test with sample data
- [ ] Handle edge cases (empty data, errors)
### Deployment Phase
- [ ] Review workflow settings (execution order, timeout, error handling)
- [ ] Activate workflow using `activateWorkflow` operation
- [ ] Monitor first executions
- [ ] Document workflow purpose and data flow
---
## Data Flow Patterns
### Linear Flow
```
Trigger → Transform → Action → End
```
**Use when**: Simple workflows with single path
### Branching Flow
```
Trigger → IF → [True Path]
└→ [False Path]
```
**Use when**: Different actions based on conditions
### Parallel Processing
```
Trigger → [Branch 1] → Merge
└→ [Branch 2] ↗
```
**Use when**: Independent operations that can run simultaneously
### Loop Pattern
```
Trigger → Split in Batches → Process → Loop (until done)
```
**Use when**: Processing large datasets in chunks
### Error Handler Pattern
```
Main Flow → [Success Path]
└→ [Error Trigger → Error Handler]
```
**Use when**: Need separate error handling workflow
---
## Common Gotchas
### 1. Webhook Data Structure
**Problem**: Can't access webhook payload data
**Solution**: Data is nested under `$json.body`
```javascript
❌ {{$json.email}}
✅ {{$json.body.email}}
```
See: n8n Expression Syntax skill
### 2. Multiple Input Items
**Problem**: Node processes all input items, but I only want one
**Solution**: Use "Execute Once" mode or process first item only
```javascript
{{$json[0].field}} // First item only
```
### 3. Authentication Issues
**Problem**: API calls failing with 401/403
**Solution**:
- Configure credentials properly
- Use the "Credentials" section, not parameters
- Test credentials before workflow activation
### 4. Node Execution Order
**Problem**: Nodes executing in unexpected order
**Solution**: Check workflow settings → Execution Order
- v0: Top-to-bottom (legacy)
- v1: Connection-based (recommended)
### 5. Expression Errors
**Problem**: Expressions showing as literal text
**Solution**: Use {{}} around expressions
- See n8n Expression Syntax skill for details
---
## Integration with Other Skills
These skills work together with Workflow Patterns:
**n8n MCP Tools Expert** - Use to:
- Find nodes for your pattern (search_nodes)
- Understand node operations (get_node)
- Create workflows (n8n_create_workflow)
- Deploy templates (n8n_deploy_template)
- Use ai_agents_guide for AI pattern guidance
**n8n Expression Syntax** - Use to:
- Write expressions in transformation nodes
- Access webhook data correctly ({{$json.body.field}})
- Reference previous nodes ({{$node["Node Name"].json.field}})
**n8n Node Configuration** - Use to:
- Configure specific operations for pattern nodes
- Understand node-specific requirements
**n8n Validation Expert** - Use to:
- Validate workflow structure
- Fix validation errors
- Ensure workflow correctness before deployment
---
## Pattern Statistics
Common workflow patterns:
**Most Common Triggers**:
1. Webhook - 35%
2. Schedule (periodic tasks) - 28%
3. Manual (testing/admin) - 22%
4. Service triggers (Slack, email, etc.) - 15%
**Most Common Transformations**:
1. Set (field mapping) - 68%
2. Code (custom logic) - 42%
3. IF (conditional routing) - 38%
4. Switch (multi-condition) - 18%
**Most Common Outputs**:
1. HTTP Request (APIs) - 45%
2. Slack - 32%
3. Database writes - 28%
4. Email - 24%
**Average Workflow Complexity**:
- Simple (3-5 nodes): 42%
- Medium (6-10 nodes): 38%
- Complex (11+ nodes): 20%
---
## Quick Start Examples
### Example 1: Simple Webhook → Slack
```
1. Webhook (path: "form-submit", POST)
2. Set (map form fields)
3. Slack (post message to #notifications)
```
### Example 2: Scheduled Report
```
1. Schedule (daily at 9 AM)
2. HTTP Request (fetch analytics)
3. Code (aggregate data)
4. Email (send formatted report)
5. Error Trigger → Slack (notify on failure)
```
### Example 3: Database Sync
```
1. Schedule (every 15 minutes)
2. Postgres (query new records)
3. IF (check if records exist)
4. MySQL (insert records)
5. Postgres (update sync timestamp)
```
### Example 4: AI Assistant
```
1. Webhook (receive chat message)
2. AI Agent
├─ OpenAI Chat Model (ai_languageModel)
├─ HTTP Request Tool (ai_tool)
├─ Database Tool (ai_tool)
└─ Window Buffer Memory (ai_memory)
3. Webhook Response (send AI reply)
```
### Example 5: API Integration
```
1. Manual Trigger (for testing)
2. HTTP Request (GET /api/users)
3. Split In Batches (process 100 at a time)
4. Set (transform user data)
5. Postgres (upsert users)
6. Loop (back to step 3 until done)
```
---
## Detailed Pattern Files
For comprehensive guidance on each pattern:
- **[webhook_processing.md](webhook_processing.md)** - Webhook patterns, data structure, response handling
- **[http_api_integration.md](http_api_integration.md)** - REST APIs, authentication, pagination, retries
- **[database_operations.md](database_operations.md)** - Queries, sync, transactions, batch processing
- **[ai_agent_workflow.md](ai_agent_workflow.md)** - AI agents, tools, memory, langchain nodes
- **[scheduled_tasks.md](scheduled_tasks.md)** - Cron schedules, reports, maintenance tasks
---
## Real Template Examples
From n8n template library:
**Template #2947**: Weather to Slack
- Pattern: Scheduled Task
- Nodes: Schedule → HTTP Request (weather API) → Set → Slack
- Complexity: Simple (4 nodes)
**Webhook Processing**: Most common pattern
- Most common: Form submissions, payment webhooks, chat integrations
**HTTP API**: Common pattern
- Most common: Data fetching, third-party integrations
**Database Operations**: Common pattern
- Most common: ETL, data sync, backup workflows
**AI Agents**: Growing in usage
- Most common: Chatbots, content generation, data analysis
Use `search_templates` and `get_template` from n8n-mcp tools to find examples!
---
## Best Practices
### ✅ Do
- Start with the simplest pattern that solves your problem
- Plan your workflow structure before building
- Use error handling on all workflows
- Test with sample data before activation
- Follow the workflow creation checklist
- Use descriptive node names
- Document complex workflows (notes field)
- Monitor workflow executions after deployment
### ❌ Don't
- Build workflows in one shot (iterate! avg 56s between edits)
- Skip validation before activation
- Ignore error scenarios
- Use complex patterns when simple ones suffice
- Hardcode credentials in parameters
- Forget to handle empty data cases
- Mix multiple patterns without clear boundaries
- Deploy without testing
---
## Summary
**Key Points**:
1. **5 core patterns** cover 90%+ of workflow use cases
2. **Webhook processing** is the most common pattern
3. Use the **workflow creation checklist** for every workflow
4. **Plan pattern** → **Select nodes** → **Build** → **Validate** → **Deploy**
5. Integrate with other skills for complete workflow development
**Next Steps**:
1. Identify your use case pattern
2. Read the detailed pattern file
3. Use n8n MCP Tools Expert to find nodes
4. Follow the workflow creation checklist
5. Use n8n Validation Expert to validate
**Related Skills**:
- n8n MCP Tools Expert - Find and configure nodes
- n8n Expression Syntax - Write expressions correctly
- n8n Validation Expert - Validate and fix errors
- n8n Node Configuration - Configure specific operations

784
.opencode/skills-off/n8n-workflow-patterns/ai_agent_workflow.md Normal file
View File

@@ -0,0 +1,784 @@
# AI Agent Workflow Pattern
**Use Case**: Build AI agents with tool access, memory, and reasoning capabilities.
---
## Pattern Structure
```
Trigger → AI Agent (Model + Tools + Memory) → [Process Response] → Output
```
**Key Characteristic**: AI-powered decision making with tool use
---
## Core AI Connection Types
n8n supports **8 AI connection types** for building agent workflows:
1. **ai_languageModel** - The LLM (OpenAI, Anthropic, etc.)
2. **ai_tool** - Functions the agent can call
3. **ai_memory** - Conversation context
4. **ai_outputParser** - Parse structured outputs
5. **ai_embedding** - Vector embeddings
6. **ai_vectorStore** - Vector database
7. **ai_document** - Document loaders
8. **ai_textSplitter** - Text chunking
---
## Core Components
### 1. Trigger
**Options**:
- **Webhook** - Chat interfaces, API calls (most common)
- **Manual** - Testing and development
- **Schedule** - Periodic AI tasks
### 2. AI Agent Node
**Purpose**: Orchestrate LLM with tools and memory
**Configuration**:
```javascript
{
agent: "conversationalAgent", // or "openAIFunctionsAgent"
promptType: "define",
text: "You are a helpful assistant that can search docs, query databases, and send emails."
}
```
**Connections**:
- **ai_languageModel input** - Connected to LLM node
- **ai_tool inputs** - Connected to tool nodes
- **ai_memory input** - Connected to memory node (optional)
### 3. Language Model
**Available providers**:
- OpenAI (GPT-4, GPT-3.5)
- Anthropic (Claude)
- Google (Gemini)
- Local models (Ollama, LM Studio)
**Example** (OpenAI Chat Model):
```javascript
{
model: "gpt-4",
temperature: 0.7,
maxTokens: 1000
}
```
### 4. Tools (ANY Node Can Be a Tool!)
**Critical insight**: Connect ANY n8n node to agent via `ai_tool` port
**Common tool types**:
- HTTP Request - Call APIs
- Database nodes - Query data
- Code - Custom functions
- Search nodes - Web/document search
- Pre-built tool nodes (Calculator, Wikipedia, etc.)
### 5. Memory (Optional but Recommended)
**Purpose**: Maintain conversation context
**Types**:
- **Buffer Memory** - Store recent messages
- **Window Buffer Memory** - Store last N messages
- **Summary Memory** - Summarize conversation
### 6. Output Processing
**Purpose**: Format AI response for delivery
**Common patterns**:
- Return directly (chat response)
- Store in database (conversation history)
- Send to communication channel (Slack, email)
---
## Common Use Cases
### 1. Conversational Chatbot
**Flow**: Webhook (chat message) → AI Agent → Webhook Response
**Example** (Customer support bot):
```
1. Webhook (path: "chat", POST)
- Receives: {user_id, message, session_id}
2. Window Buffer Memory (load context by session_id)
3. AI Agent
├─ OpenAI Chat Model (gpt-4)
├─ HTTP Request Tool (search knowledge base)
├─ Database Tool (query customer orders)
└─ Window Buffer Memory (conversation context)
4. Code (format response)
5. Webhook Response (send reply)
```
**AI Agent prompt**:
```
You are a customer support assistant.
You can:
1. Search the knowledge base for answers
2. Look up customer orders
3. Provide shipping information
Be helpful and professional.
```
### 2. Document Q&A
**Flow**: Upload docs → Embed → Store → Query with AI
**Example** (Internal documentation assistant):
```
Setup Phase (run once):
1. Read Files (load documentation)
2. Text Splitter (chunk into paragraphs)
3. Embeddings (OpenAI Embeddings)
4. Vector Store (Pinecone/Qdrant) (store vectors)
Query Phase (recurring):
1. Webhook (receive question)
2. AI Agent
├─ OpenAI Chat Model (gpt-4)
├─ Vector Store Tool (search similar docs)
└─ Buffer Memory (context)
3. Webhook Response (answer with citations)
```
### 3. Data Analysis Assistant
**Flow**: Request → AI Agent (with data tools) → Analysis → Visualization
**Example** (SQL analyst agent):
```
1. Webhook (data question: "What were sales last month?")
2. AI Agent
├─ OpenAI Chat Model (gpt-4)
├─ Postgres Tool (execute queries)
└─ Code Tool (data analysis)
3. Code (generate visualization data)
4. Webhook Response (answer + chart data)
```
**Postgres Tool Configuration**:
```javascript
{
name: "query_database",
description: "Execute SQL queries to analyze sales data. Use SELECT queries only.",
// Node executes AI-generated SQL
}
```
### 4. Workflow Automation Agent
**Flow**: Command → AI Agent → Execute actions → Report
**Example** (DevOps assistant):
```
1. Slack (slash command: /deploy production)
2. AI Agent
├─ OpenAI Chat Model (gpt-4)
├─ HTTP Request Tool (GitHub API)
├─ HTTP Request Tool (Deploy API)
└─ Postgres Tool (deployment logs)
3. Agent actions:
- Check if tests passed
- Create deployment
- Log deployment
- Notify team
4. Slack (deployment status)
```
### 5. Email Processing Agent
**Flow**: Email received → AI Agent → Categorize → Route → Respond
**Example** (Support ticket router):
```
1. Email Trigger (new support email)
2. AI Agent
├─ OpenAI Chat Model (gpt-4)
├─ Vector Store Tool (search similar tickets)
└─ HTTP Request Tool (create Jira ticket)
3. Agent actions:
- Categorize urgency (low/medium/high)
- Find similar past tickets
- Create ticket in appropriate project
- Draft response
4. Email (send auto-response)
5. Slack (notify assigned team)
```
---
## Tool Configuration
### Making ANY Node an AI Tool
**Critical concept**: Any n8n node can become an AI tool!
**Requirements**:
1. Connect node to AI Agent via `ai_tool` port (NOT main port)
2. Configure tool name and description
3. Define input schema (optional)
**Example** (HTTP Request as tool):
```javascript
{
// Tool metadata (for AI)
name: "search_github_issues",
description: "Search GitHub issues by keyword. Returns issue titles and URLs.",
// HTTP Request configuration
method: "GET",
url: "https://api.github.com/search/issues",
sendQuery: true,
queryParameters: {
"q": "={{$json.query}} repo:{{$json.repo}}",
"per_page": "5"
}
}
```
**How it works**:
1. AI Agent sees tool: `search_github_issues(query, repo)`
2. AI decides to use it: `search_github_issues("bug", "n8n-io/n8n")`
3. n8n executes HTTP Request with parameters
4. Result returned to AI Agent
5. AI Agent processes result and responds
### Pre-built Tool Nodes
**Available in @n8n/n8n-nodes-langchain**:
- **Calculator Tool** - Math operations
- **Wikipedia Tool** - Wikipedia search
- **Serper Tool** - Google search
- **Wolfram Alpha Tool** - Computational knowledge
- **Custom Tool** - Define with Code node
**Example** (Calculator Tool):
```
AI Agent
├─ OpenAI Chat Model
└─ Calculator Tool (ai_tool connection)
User: "What's 15% of 2,847?"
AI: *uses calculator tool* → "426.05"
```
### Database as Tool
**Pattern**: Postgres/MySQL node connected as ai_tool
**Configuration**:
```javascript
{
// Tool metadata
name: "query_customers",
description: "Query customer database. Use SELECT queries to find customer information by email, name, or ID.",
// Postgres config
operation: "executeQuery",
query: "={{$json.sql}}", // AI provides SQL
// Security: Use read-only database user!
}
```
**Safety**: Create read-only DB user for AI tools!
```sql
CREATE USER ai_readonly WITH PASSWORD 'secure_password';
GRANT SELECT ON customers, orders TO ai_readonly;
-- NO INSERT, UPDATE, DELETE access
```
### Code Node as Tool
**Pattern**: Custom Python/JavaScript function
**Example** (Data processor):
```javascript
// Tool metadata
{
name: "process_csv",
description: "Process CSV data and return statistics. Input: csv_string"
}
// Code node
const csv = $input.first().json.csv_string;
const lines = csv.split('\n');
const data = lines.slice(1).map(line => line.split(','));
return [{
json: {
row_count: data.length,
columns: lines[0].split(','),
summary: {
// Calculate statistics
}
}
}];
```
---
## Memory Configuration
### Buffer Memory
**Stores all messages** (until cleared)
```javascript
{
memoryType: "bufferMemory",
sessionKey: "={{$json.body.user_id}}" // Per-user memory
}
```
### Window Buffer Memory
**Stores last N messages** (recommended)
```javascript
{
memoryType: "windowBufferMemory",
sessionKey: "={{$json.body.session_id}}",
contextWindowLength: 10 // Last 10 messages
}
```
### Summary Memory
**Summarizes old messages** (for long conversations)
```javascript
{
memoryType: "summaryMemory",
sessionKey: "={{$json.body.session_id}}",
maxTokenLimit: 2000
}
```
**How it works**:
1. Conversation grows beyond limit
2. AI summarizes old messages
3. Summary stored, old messages discarded
4. Saves tokens while maintaining context
---
## Agent Types
### 1. Conversational Agent
**Best for**: General chat, customer support
**Features**:
- Natural conversation flow
- Memory integration
- Tool use with reasoning
**When to use**: Most common use case
### 2. OpenAI Functions Agent
**Best for**: Tool-heavy workflows, structured outputs
**Features**:
- Optimized for function calling
- Better tool selection
- Structured responses
**When to use**: Multiple tools, need reliable tool calling
### 3. ReAct Agent
**Best for**: Step-by-step reasoning
**Features**:
- Think → Act → Observe loop
- Visible reasoning process
- Good for debugging
**When to use**: Complex multi-step tasks
---
## Prompt Engineering for Agents
### System Prompt Structure
```
You are a [ROLE].
You can:
- [CAPABILITY 1]
- [CAPABILITY 2]
- [CAPABILITY 3]
Guidelines:
- [GUIDELINE 1]
- [GUIDELINE 2]
Format:
- [OUTPUT FORMAT]
```
### Example (Customer Support)
```
You are a customer support assistant for Acme Corp.
You can:
- Search the knowledge base for answers
- Look up customer orders and shipping status
- Create support tickets for complex issues
Guidelines:
- Be friendly and professional
- If you don't know something, say so and offer to create a ticket
- Always verify customer identity before sharing order details
Format:
- Keep responses concise
- Use bullet points for multiple items
- Include relevant links when available
```
### Example (Data Analyst)
```
You are a data analyst assistant with access to the company database.
You can:
- Query sales, customer, and product data
- Perform data analysis and calculations
- Generate summary statistics
Guidelines:
- Write efficient SQL queries (always use LIMIT)
- Explain your analysis methodology
- Highlight important trends or anomalies
- Use read-only queries (SELECT only)
Format:
- Provide numerical answers with context
- Include query used (for transparency)
- Suggest follow-up analyses when relevant
```
---
## Error Handling
### Pattern 1: Tool Execution Errors
```
AI Agent (continueOnFail on tool nodes)
→ IF (tool error occurred)
└─ Code (log error)
└─ Webhook Response (user-friendly error)
```
### Pattern 2: LLM API Errors
```
Main Workflow:
AI Agent → Process Response
Error Workflow:
Error Trigger
→ IF (rate limit error)
└─ Wait → Retry
→ ELSE
└─ Notify Admin
```
### Pattern 3: Invalid Tool Outputs
```javascript
// Code node - validate tool output
const result = $input.first().json;
if (!result || !result.data) {
throw new Error('Tool returned invalid data');
}
return [{ json: result }];
```
---
## Performance Optimization
### 1. Choose Right Model
```
Fast & cheap: GPT-3.5-turbo, Claude 3 Haiku
Balanced: GPT-4, Claude 3 Sonnet
Powerful: GPT-4-turbo, Claude 3 Opus
```
### 2. Limit Context Window
```javascript
{
memoryType: "windowBufferMemory",
contextWindowLength: 5 // Only last 5 messages
}
```
### 3. Optimize Tool Descriptions
```javascript
// ❌ Vague
description: "Search for things"
// ✅ Clear and concise
description: "Search GitHub issues by keyword and repository. Returns top 5 matching issues with titles and URLs."
```
### 4. Cache Embeddings
For document Q&A, embed documents once:
```
Setup (run once):
Documents → Embed → Store in Vector DB
Query (fast):
Question → Search Vector DB → AI Agent
```
### 5. Async Tools for Slow Operations
```
AI Agent → [Queue slow tool request]
→ Return immediate response
→ [Background: Execute tool + notify when done]
```
---
## Security Considerations
### 1. Read-Only Database Tools
```sql
-- Create limited user for AI tools
CREATE USER ai_agent_ro WITH PASSWORD 'secure';
GRANT SELECT ON public.* TO ai_agent_ro;
-- NO write access!
```
### 2. Validate Tool Inputs
```javascript
// Code node - validate before execution
const query = $json.query;
if (query.toLowerCase().includes('drop ') ||
query.toLowerCase().includes('delete ') ||
query.toLowerCase().includes('update ')) {
throw new Error('Invalid query - write operations not allowed');
}
```
### 3. Rate Limiting
```
Webhook → IF (check user rate limit)
├─ [Within limit] → AI Agent
└─ [Exceeded] → Error (429 Too Many Requests)
```
### 4. Sanitize User Input
```javascript
// Code node
const userInput = $json.body.message
.trim()
.substring(0, 1000); // Max 1000 chars
return [{ json: { sanitized: userInput } }];
```
### 5. Monitor Tool Usage
```
AI Agent → Log Tool Calls
→ IF (suspicious pattern)
└─ Alert Admin + Pause Agent
```
---
## Testing AI Agents
### 1. Start with Manual Trigger
Replace webhook with manual trigger:
```
Manual Trigger
→ Set (mock user input)
→ AI Agent
→ Code (log output)
```
### 2. Test Tools Independently
Before connecting to agent:
```
Manual Trigger → Tool Node → Verify output format
```
### 3. Test with Standard Questions
Create test suite:
```
1. "Hello" - Test basic response
2. "Search for bug reports" - Test tool calling
3. "What did I ask before?" - Test memory
4. Invalid input - Test error handling
```
### 4. Monitor Token Usage
```javascript
// Code node - log token usage
console.log('Input tokens:', $node['AI Agent'].json.usage.input_tokens);
console.log('Output tokens:', $node['AI Agent'].json.usage.output_tokens);
```
### 5. Test Edge Cases
- Empty input
- Very long input
- Tool returns no results
- Tool returns error
- Multiple tool calls in sequence
---
## Common Gotchas
### 1. ❌ Wrong: Connecting tools to main port
```
HTTP Request → AI Agent // Won't work as tool!
```
### ✅ Correct: Use ai_tool connection type
```
HTTP Request --[ai_tool]--> AI Agent
```
### 2. ❌ Wrong: Vague tool descriptions
```
description: "Get data" // AI won't know when to use this
```
### ✅ Correct: Specific descriptions
```
description: "Query customer orders by email address. Returns order ID, status, and shipping info."
```
### 3. ❌ Wrong: No memory for conversations
```
Every message is standalone - no context!
```
### ✅ Correct: Add memory
```
Window Buffer Memory --[ai_memory]--> AI Agent
```
### 4. ❌ Wrong: Giving AI write access
```
Postgres (full access) as tool // AI could DELETE data!
```
### ✅ Correct: Read-only access
```
Postgres (read-only user) as tool // Safe
```
### 5. ❌ Wrong: Unbounded tool responses
```
Tool returns 10MB of data → exceeds token limit
```
### ✅ Correct: Limit tool output
```javascript
{
query: "SELECT * FROM table LIMIT 10" // Only 10 rows
}
```
---
## Real Template Examples
From n8n template library (234 AI templates):
**Simple Chatbot**:
```
Webhook → AI Agent (GPT-4 + Memory) → Webhook Response
```
**Document Q&A**:
```
Setup: Files → Embed → Vector Store
Query: Webhook → AI Agent (GPT-4 + Vector Store Tool) → Response
```
**SQL Analyst**:
```
Webhook → AI Agent (GPT-4 + Postgres Tool) → Format → Response
```
Use `search_templates({query: "ai agent"})` to find more!
---
## Checklist for AI Agent Workflows
### Planning
- [ ] Define agent purpose and capabilities
- [ ] List required tools (APIs, databases, etc.)
- [ ] Design conversation flow
- [ ] Plan memory strategy (per-user, per-session)
- [ ] Consider token costs
### Implementation
- [ ] Choose appropriate LLM model
- [ ] Write clear system prompt
- [ ] Connect tools via ai_tool ports (NOT main)
- [ ] Add tool descriptions
- [ ] Configure memory (Window Buffer recommended)
- [ ] Test each tool independently
### Security
- [ ] Use read-only database access for tools
- [ ] Validate tool inputs
- [ ] Sanitize user inputs
- [ ] Add rate limiting
- [ ] Monitor for abuse
### Testing
- [ ] Test with diverse inputs
- [ ] Verify tool calling works
- [ ] Check memory persistence
- [ ] Test error scenarios
- [ ] Monitor token usage and costs
### Deployment
- [ ] Add error handling
- [ ] Set up logging
- [ ] Monitor performance
- [ ] Set cost alerts
- [ ] Document agent capabilities
---
## Summary
**Key Points**:
1. **8 AI connection types** - Use ai_tool for tools, ai_memory for context
2. **ANY node can be a tool** - Connect to ai_tool port
3. **Memory is essential** for conversations (Window Buffer recommended)
4. **Tool descriptions matter** - AI uses them to decide when to call tools
5. **Security first** - Read-only database access, validate inputs
**Pattern**: Trigger → AI Agent (Model + Tools + Memory) → Output
**Related**:
- [webhook_processing.md](webhook_processing.md) - Receiving chat messages
- [http_api_integration.md](http_api_integration.md) - Tools that call APIs
- [database_operations.md](database_operations.md) - Database tools for agents

785
.opencode/skills-off/n8n-workflow-patterns/database_operations.md Normal file
View File

@@ -0,0 +1,785 @@
# Database Operations Pattern
**Use Case**: Read, write, sync, and manage database data in workflows.
---
## Pattern Structure
```
Trigger → [Query/Read] → [Transform] → [Write/Update] → [Verify/Log]
```
**Key Characteristic**: Data persistence and synchronization
---
## Core Components
### 1. Trigger
**Options**:
- **Schedule** - Periodic sync/maintenance (most common)
- **Webhook** - Event-driven writes
- **Manual** - One-time operations
### 2. Database Read Nodes
**Supported databases**:
- Postgres
- MySQL
- MongoDB
- Microsoft SQL
- SQLite
- Redis
- And more via community nodes
### 3. Transform
**Purpose**: Map between different database schemas or formats
**Typical nodes**:
- **Set** - Field mapping
- **Code** - Complex transformations
- **Merge** - Combine data from multiple sources
### 4. Database Write Nodes
**Operations**:
- INSERT - Create new records
- UPDATE - Modify existing records
- UPSERT - Insert or update
- DELETE - Remove records
### 5. Verification
**Purpose**: Confirm operations succeeded
**Methods**:
- Query to verify records
- Count rows affected
- Log results
---
## Common Use Cases
### 1. Data Synchronization
**Flow**: Schedule → Read Source DB → Transform → Write Target DB → Log
**Example** (Postgres to MySQL sync):
```
1. Schedule (every 15 minutes)
2. Postgres (SELECT * FROM users WHERE updated_at > {{$json.last_sync}})
3. IF (check if records exist)
4. Set (map Postgres schema to MySQL schema)
5. MySQL (INSERT or UPDATE users)
6. Postgres (UPDATE sync_log SET last_sync = NOW())
7. Slack (notify: "Synced X users")
```
**Incremental sync query**:
```sql
SELECT *
FROM users
WHERE updated_at > $1
ORDER BY updated_at ASC
LIMIT 1000
```
**Parameters**:
```javascript
{
"parameters": [
"={{$node['Get Last Sync'].json.last_sync}}"
]
}
```
### 2. ETL (Extract, Transform, Load)
**Flow**: Extract from multiple sources → Transform → Load into warehouse
**Example** (Consolidate data):
```
1. Schedule (daily at 2 AM)
2. [Parallel branches]
├─ Postgres (SELECT orders)
├─ MySQL (SELECT customers)
└─ MongoDB (SELECT products)
3. Merge (combine all data)
4. Code (transform to warehouse schema)
5. Postgres (warehouse - INSERT into fact_sales)
6. Email (send summary report)
```
### 3. Data Validation & Cleanup
**Flow**: Schedule → Query → Validate → Update/Delete invalid records
**Example** (Clean orphaned records):
```
1. Schedule (weekly)
2. Postgres (SELECT users WHERE email IS NULL OR email = '')
3. IF (invalid records exist)
4. Postgres (UPDATE users SET status='inactive' WHERE email IS NULL)
5. Postgres (DELETE FROM users WHERE created_at < NOW() - INTERVAL '1 year' AND status='inactive')
6. Slack (alert: "Cleaned X invalid records")
```
### 4. Backup & Archive
**Flow**: Schedule → Query → Export → Store
**Example** (Archive old records):
```
1. Schedule (monthly)
2. Postgres (SELECT * FROM orders WHERE created_at < NOW() - INTERVAL '2 years')
3. Code (convert to JSON)
4. Write File (save to archive.json)
5. Google Drive (upload archive)
6. Postgres (DELETE FROM orders WHERE created_at < NOW() - INTERVAL '2 years')
```
### 5. Real-time Data Updates
**Flow**: Webhook → Parse → Update Database
**Example** (Update user status):
```
1. Webhook (receive status update)
2. Postgres (UPDATE users SET status = {{$json.body.status}} WHERE id = {{$json.body.user_id}})
3. IF (rows affected > 0)
4. Redis (SET user:{{$json.body.user_id}}:status {{$json.body.status}})
5. Webhook Response ({"success": true})
```
---
## Database Node Configuration
### Postgres
#### SELECT Query
```javascript
{
operation: "executeQuery",
query: "SELECT id, name, email FROM users WHERE created_at > $1 LIMIT $2",
parameters: [
"={{$json.since_date}}",
"100"
]
}
```
#### INSERT
```javascript
{
operation: "insert",
table: "users",
columns: "id, name, email, created_at",
values: [
{
id: "={{$json.id}}",
name: "={{$json.name}}",
email: "={{$json.email}}",
created_at: "={{$now}}"
}
]
}
```
#### UPDATE
```javascript
{
operation: "update",
table: "users",
updateKey: "id",
columns: "name, email, updated_at",
values: {
id: "={{$json.id}}",
name: "={{$json.name}}",
email: "={{$json.email}}",
updated_at: "={{$now}}"
}
}
```
#### UPSERT (INSERT ... ON CONFLICT)
```javascript
{
operation: "executeQuery",
query: `
INSERT INTO users (id, name, email)
VALUES ($1, $2, $3)
ON CONFLICT (id)
DO UPDATE SET name = $2, email = $3, updated_at = NOW()
`,
parameters: [
"={{$json.id}}",
"={{$json.name}}",
"={{$json.email}}"
]
}
```
### MySQL
#### SELECT with JOIN
```javascript
{
operation: "executeQuery",
query: `
SELECT u.id, u.name, o.order_id, o.total
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
WHERE u.created_at > ?
`,
parameters: [
"={{$json.since_date}}"
]
}
```
#### Bulk INSERT
```javascript
{
operation: "insert",
table: "orders",
columns: "user_id, total, status",
values: $json.orders // Array of objects
}
```
### MongoDB
#### Find Documents
```javascript
{
operation: "find",
collection: "users",
query: JSON.stringify({
created_at: { $gt: new Date($json.since_date) },
status: "active"
}),
limit: 100
}
```
#### Insert Document
```javascript
{
operation: "insert",
collection: "users",
document: JSON.stringify({
name: $json.name,
email: $json.email,
created_at: new Date()
})
}
```
#### Update Document
```javascript
{
operation: "update",
collection: "users",
query: JSON.stringify({ _id: $json.user_id }),
update: JSON.stringify({
$set: {
status: $json.status,
updated_at: new Date()
}
})
}
```
---
## Batch Processing
### Pattern 1: Split In Batches
**Use when**: Processing large datasets to avoid memory issues
```
Postgres (SELECT 10000 records)
→ Split In Batches (100 items per batch)
→ Transform
→ MySQL (write batch)
→ Loop (until all processed)
```
### Pattern 2: Paginated Queries
**Use when**: Database has millions of records
```
Set (initialize: offset=0, limit=1000)
→ Loop Start
→ Postgres (SELECT * FROM large_table LIMIT {{$json.limit}} OFFSET {{$json.offset}})
→ IF (records returned)
├─ Process records
├─ Set (increment offset by 1000)
└─ Loop back
└─ [No records] → End
```
**Query**:
```sql
SELECT * FROM large_table
ORDER BY id
LIMIT $1 OFFSET $2
```
### Pattern 3: Cursor-Based Pagination
**Better performance for large datasets**:
```
Set (initialize: last_id=0)
→ Loop Start
→ Postgres (SELECT * FROM table WHERE id > {{$json.last_id}} ORDER BY id LIMIT 1000)
→ IF (records returned)
├─ Process records
├─ Code (get max id from batch)
└─ Loop back
└─ [No records] → End
```
**Query**:
```sql
SELECT * FROM table
WHERE id > $1
ORDER BY id ASC
LIMIT 1000
```
---
## Transaction Handling
### Pattern 1: BEGIN/COMMIT/ROLLBACK
**For databases that support transactions**:
```javascript
// Node 1: Begin Transaction
{
operation: "executeQuery",
query: "BEGIN"
}
// Node 2-N: Your operations
{
operation: "executeQuery",
query: "INSERT INTO ...",
continueOnFail: true
}
// Node N+1: Commit or Rollback
{
operation: "executeQuery",
query: "={{$node['Operation'].json.error ? 'ROLLBACK' : 'COMMIT'}}"
}
```
### Pattern 2: Atomic Operations
**Use database features for atomicity**:
```sql
-- Upsert example (atomic)
INSERT INTO inventory (product_id, quantity)
VALUES ($1, $2)
ON CONFLICT (product_id)
DO UPDATE SET quantity = inventory.quantity + $2
```
### Pattern 3: Error Rollback
**Manual rollback on error**:
```
Try Operations:
Postgres (INSERT orders)
MySQL (INSERT order_items)
Error Trigger:
Postgres (DELETE FROM orders WHERE id = {{$json.order_id}})
MySQL (DELETE FROM order_items WHERE order_id = {{$json.order_id}})
```
---
## Data Transformation
### Schema Mapping
```javascript
// Code node - map schemas
const sourceData = $input.all();
return sourceData.map(item => ({
json: {
// Source → Target mapping
user_id: item.json.id,
full_name: `${item.json.first_name} ${item.json.last_name}`,
email_address: item.json.email,
registration_date: new Date(item.json.created_at).toISOString(),
// Computed fields
is_premium: item.json.plan_type === 'pro',
// Default values
status: item.json.status || 'active'
}
}));
```
### Data Type Conversions
```javascript
// Code node - convert data types
return $input.all().map(item => ({
json: {
// String to number
user_id: parseInt(item.json.user_id),
// String to date
created_at: new Date(item.json.created_at),
// Number to boolean
is_active: item.json.active === 1,
// JSON string to object
metadata: JSON.parse(item.json.metadata || '{}'),
// Null handling
email: item.json.email || null
}
}));
```
### Aggregation
```javascript
// Code node - aggregate data
const items = $input.all();
const summary = items.reduce((acc, item) => {
const date = item.json.created_at.split('T')[0];
if (!acc[date]) {
acc[date] = { count: 0, total: 0 };
}
acc[date].count++;
acc[date].total += item.json.amount;
return acc;
}, {});
return Object.entries(summary).map(([date, data]) => ({
json: {
date,
count: data.count,
total: data.total,
average: data.total / data.count
}
}));
```
---
## Performance Optimization
### 1. Use Indexes
Ensure database has proper indexes:
```sql
-- Add index for sync queries
CREATE INDEX idx_users_updated_at ON users(updated_at);
-- Add index for lookups
CREATE INDEX idx_orders_user_id ON orders(user_id);
```
### 2. Limit Result Sets
Always use LIMIT:
```sql
-- ✅ Good
SELECT * FROM large_table
WHERE created_at > $1
LIMIT 1000
-- ❌ Bad (unbounded)
SELECT * FROM large_table
WHERE created_at > $1
```
### 3. Use Prepared Statements
Parameterized queries are faster:
```javascript
// ✅ Good - prepared statement
{
query: "SELECT * FROM users WHERE id = $1",
parameters: ["={{$json.id}}"]
}
// ❌ Bad - string concatenation
{
query: "SELECT * FROM users WHERE id = '={{$json.id}}'"
}
```
### 4. Batch Writes
Write multiple records at once:
```javascript
// ✅ Good - batch insert
{
operation: "insert",
table: "orders",
values: $json.items // Array of 100 items
}
// ❌ Bad - individual inserts in loop
// 100 separate INSERT statements
```
### 5. Connection Pooling
Configure in credentials:
```javascript
{
host: "db.example.com",
database: "mydb",
user: "user",
password: "pass",
// Connection pool settings
min: 2,
max: 10,
idleTimeoutMillis: 30000
}
```
---
## Error Handling
### Pattern 1: Check Rows Affected
```
Database Operation (UPDATE users...)
→ IF ({{$json.rowsAffected === 0}})
└─ Alert: "No rows updated - record not found"
```
### Pattern 2: Constraint Violations
```javascript
// Database operation with continueOnFail: true
{
operation: "insert",
continueOnFail: true
}
// Next node: Check for errors
IF ({{$json.error !== undefined}})
→ IF ({{$json.error.includes('duplicate key')}})
└─ Log: "Record already exists - skipping"
→ ELSE
└─ Alert: "Database error: {{$json.error}}"
```
### Pattern 3: Rollback on Error
```
Try Operations:
→ Database Write 1
→ Database Write 2
→ Database Write 3
Error Trigger:
→ Rollback Operations
→ Alert Admin
```
---
## Security Best Practices
### 1. Use Parameterized Queries (Prevent SQL Injection)
```javascript
// ✅ SAFE - parameterized
{
query: "SELECT * FROM users WHERE email = $1",
parameters: ["={{$json.email}}"]
}
// ❌ DANGEROUS - SQL injection risk
{
query: "SELECT * FROM users WHERE email = '={{$json.email}}'"
}
```
### 2. Least Privilege Access
**Create dedicated workflow user**:
```sql
-- ✅ Good - limited permissions
CREATE USER n8n_workflow WITH PASSWORD 'secure_password';
GRANT SELECT, INSERT, UPDATE ON orders TO n8n_workflow;
GRANT SELECT ON users TO n8n_workflow;
-- ❌ Bad - too much access
GRANT ALL PRIVILEGES TO n8n_workflow;
```
### 3. Validate Input Data
```javascript
// Code node - validate before write
const email = $json.email;
const name = $json.name;
// Validation
if (!email || !email.includes('@')) {
throw new Error('Invalid email address');
}
if (!name || name.length < 2) {
throw new Error('Invalid name');
}
// Sanitization
return [{
json: {
email: email.toLowerCase().trim(),
name: name.trim()
}
}];
```
### 4. Encrypt Sensitive Data
```javascript
// Code node - encrypt before storage
const crypto = require('crypto');
const algorithm = 'aes-256-cbc';
const key = Buffer.from($credentials.encryptionKey, 'hex');
const iv = crypto.randomBytes(16);
const cipher = crypto.createCipheriv(algorithm, key, iv);
let encrypted = cipher.update($json.sensitive_data, 'utf8', 'hex');
encrypted += cipher.final('hex');
return [{
json: {
encrypted_data: encrypted,
iv: iv.toString('hex')
}
}];
```
---
## Common Gotchas
### 1. ❌ Wrong: Unbounded queries
```sql
SELECT * FROM large_table -- Could return millions
```
### ✅ Correct: Use LIMIT
```sql
SELECT * FROM large_table
ORDER BY created_at DESC
LIMIT 1000
```
### 2. ❌ Wrong: String concatenation in queries
```javascript
query: "SELECT * FROM users WHERE id = '{{$json.id}}'"
```
### ✅ Correct: Parameterized queries
```javascript
query: "SELECT * FROM users WHERE id = $1",
parameters: ["={{$json.id}}"]
```
### 3. ❌ Wrong: No transaction for multi-step operations
```
INSERT into orders
INSERT into order_items // Fails → orphaned order record
```
### ✅ Correct: Use transaction
```
BEGIN
INSERT into orders
INSERT into order_items
COMMIT (or ROLLBACK on error)
```
### 4. ❌ Wrong: Processing all items at once
```
SELECT 1000000 records → Process all → OOM error
```
### ✅ Correct: Batch processing
```
SELECT records → Split In Batches (1000) → Process → Loop
```
---
## Real Template Examples
From n8n template library (456 database templates):
**Data Sync**:
```
Schedule → Postgres (SELECT new records) → Transform → MySQL (INSERT)
```
**ETL Pipeline**:
```
Schedule → [Multiple DB reads] → Merge → Transform → Warehouse (INSERT)
```
**Backup**:
```
Schedule → Postgres (SELECT all) → JSON → Google Drive (upload)
```
Use `search_templates({query: "database"})` to find more!
---
## Checklist for Database Workflows
### Planning
- [ ] Identify source and target databases
- [ ] Understand schema differences
- [ ] Plan transformation logic
- [ ] Consider batch size for large datasets
- [ ] Design error handling strategy
### Implementation
- [ ] Use parameterized queries (never concatenate)
- [ ] Add LIMIT to all SELECT queries
- [ ] Use appropriate operation (INSERT/UPDATE/UPSERT)
- [ ] Configure credentials properly
- [ ] Test with small dataset first
### Performance
- [ ] Add database indexes for queries
- [ ] Use batch operations
- [ ] Implement pagination for large datasets
- [ ] Configure connection pooling
- [ ] Monitor query execution times
### Security
- [ ] Use parameterized queries (SQL injection prevention)
- [ ] Least privilege database user
- [ ] Validate and sanitize input
- [ ] Encrypt sensitive data
- [ ] Never log sensitive data
### Reliability
- [ ] Add transaction handling if needed
- [ ] Check rows affected
- [ ] Handle constraint violations
- [ ] Implement retry logic
- [ ] Add Error Trigger workflow
---
## Summary
**Key Points**:
1. **Always use parameterized queries** (prevent SQL injection)
2. **Batch processing** for large datasets
3. **Transaction handling** for multi-step operations
4. **Limit result sets** to avoid memory issues
5. **Validate input data** before writes
**Pattern**: Trigger → Query → Transform → Write → Verify
**Related**:
- [http_api_integration.md](http_api_integration.md) - Fetching data to store in DB
- [scheduled_tasks.md](scheduled_tasks.md) - Periodic database maintenance

734
.opencode/skills-off/n8n-workflow-patterns/http_api_integration.md Normal file
View File

@@ -0,0 +1,734 @@
# HTTP API Integration Pattern
**Use Case**: Fetch data from REST APIs, transform it, and use it in workflows.
---
## Pattern Structure
```
Trigger → HTTP Request → [Transform] → [Action] → [Error Handler]
```
**Key Characteristic**: External data fetching with error handling
---
## Core Components
### 1. Trigger
**Options**:
- **Schedule** - Periodic fetching (most common)
- **Webhook** - Triggered by external event
- **Manual** - On-demand execution
### 2. HTTP Request Node
**Purpose**: Call external REST APIs
**Configuration**:
```javascript
{
method: "GET", // GET, POST, PUT, DELETE, PATCH
url: "https://api.example.com/users",
authentication: "predefinedCredentialType",
sendQuery: true,
queryParameters: {
"page": "={{$json.page}}",
"limit": "100"
},
sendHeaders: true,
headerParameters: {
"Accept": "application/json",
"X-API-Version": "v1"
}
}
```
### 3. Response Processing
**Purpose**: Extract and transform API response data
**Typical flow**:
```
HTTP Request → Code (parse) → Set (map fields) → Action
```
### 4. Action
**Common actions**:
- Store in database
- Send to another API
- Create notifications
- Update spreadsheet
### 5. Error Handler
**Purpose**: Handle API failures gracefully
**Error Trigger Workflow**:
```
Error Trigger → Log Error → Notify Admin → Retry Logic (optional)
```
---
## Common Use Cases
### 1. Data Fetching & Storage
**Flow**: Schedule → HTTP Request → Transform → Database
**Example** (Fetch GitHub issues):
```
1. Schedule (every hour)
2. HTTP Request
- Method: GET
- URL: https://api.github.com/repos/owner/repo/issues
- Auth: Bearer Token
- Query: state=open
3. Code (filter by labels)
4. Set (map to database schema)
5. Postgres (upsert issues)
```
**Response Handling**:
```javascript
// Code node - filter issues
const issues = $input.all();
return issues
.filter(item => item.json.labels.some(l => l.name === 'bug'))
.map(item => ({
json: {
id: item.json.id,
title: item.json.title,
created_at: item.json.created_at
}
}));
```
### 2. API to API Integration
**Flow**: Trigger → Fetch from API A → Transform → Send to API B
**Example** (Jira to Slack):
```
1. Schedule (every 15 minutes)
2. HTTP Request (GET Jira tickets updated today)
3. IF (check if tickets exist)
4. Set (format for Slack)
5. HTTP Request (POST to Slack webhook)
```
### 3. Data Enrichment
**Flow**: Trigger → Fetch base data → Call enrichment API → Combine → Store
**Example** (Enrich contacts with company data):
```
1. Postgres (SELECT new contacts)
2. Code (extract company domains)
3. HTTP Request (call Clearbit API for each domain)
4. Set (combine contact + company data)
5. Postgres (UPDATE contacts with enrichment)
```
### 4. Monitoring & Alerting
**Flow**: Schedule → Check API health → IF unhealthy → Alert
**Example** (API health check):
```
1. Schedule (every 5 minutes)
2. HTTP Request (GET /health endpoint)
3. IF (status !== 200 OR response time > 2000ms)
4. Slack (alert #ops-team)
5. PagerDuty (create incident)
```
### 5. Batch Processing
**Flow**: Trigger → Fetch large dataset → Split in Batches → Process → Loop
**Example** (Process all users):
```
1. Manual Trigger
2. HTTP Request (GET /api/users?limit=1000)
3. Split In Batches (100 items per batch)
4. HTTP Request (POST /api/process for each batch)
5. Wait (2 seconds between batches - rate limiting)
6. Loop (back to step 4 until all processed)
```
---
## Authentication Methods
### 1. None (Public APIs)
```javascript
{
authentication: "none"
}
```
### 2. Bearer Token (Most Common)
**Setup**: Create credential
```javascript
{
authentication: "predefinedCredentialType",
nodeCredentialType: "httpHeaderAuth",
headerAuth: {
name: "Authorization",
value: "Bearer YOUR_TOKEN"
}
}
```
**Access in workflow**:
```javascript
{
authentication: "predefinedCredentialType",
nodeCredentialType: "httpHeaderAuth"
}
```
### 3. API Key (Header or Query)
**Header auth**:
```javascript
{
sendHeaders: true,
headerParameters: {
"X-API-Key": "={{$credentials.apiKey}}"
}
}
```
**Query auth**:
```javascript
{
sendQuery: true,
queryParameters: {
"api_key": "={{$credentials.apiKey}}"
}
}
```
### 4. Basic Auth
**Setup**: Create "Basic Auth" credential
```javascript
{
authentication: "predefinedCredentialType",
nodeCredentialType: "httpBasicAuth"
}
```
### 5. OAuth2
**Setup**: Create OAuth2 credential with:
- Authorization URL
- Token URL
- Client ID
- Client Secret
- Scopes
```javascript
{
authentication: "predefinedCredentialType",
nodeCredentialType: "oAuth2Api"
}
```
---
## Handling API Responses
### Success Response (200-299)
**Default**: Data flows to next node
**Access response**:
```javascript
// Entire response
{{$json}}
// Specific fields
{{$json.data.id}}
{{$json.results[0].name}}
```
### Pagination
#### Pattern 1: Offset-based
```
1. Set (initialize: page=1, has_more=true)
2. HTTP Request (GET /api/items?page={{$json.page}})
3. Code (check if more pages)
4. IF (has_more === true)
└→ Set (increment page) → Loop to step 2
```
**Code node** (check pagination):
```javascript
const items = $input.first().json;
const currentPage = $json.page || 1;
return [{
json: {
items: items.results,
page: currentPage + 1,
has_more: items.next !== null
}
}];
```
#### Pattern 2: Cursor-based
```
1. HTTP Request (GET /api/items)
2. Code (extract next_cursor)
3. IF (next_cursor exists)
└→ Set (cursor={{$json.next_cursor}}) → Loop to step 1
```
#### Pattern 3: Link Header
```javascript
// Code node - parse Link header
const linkHeader = $input.first().json.headers['link'];
const hasNext = linkHeader && linkHeader.includes('rel="next"');
return [{
json: {
items: $input.first().json.body,
has_next: hasNext,
next_url: hasNext ? parseNextUrl(linkHeader) : null
}
}];
```
### Error Responses (400-599)
**Configure HTTP Request**:
```javascript
{
continueOnFail: true, // Don't stop workflow on error
ignoreResponseCode: true // Get response even on error
}
```
**Handle errors**:
```
HTTP Request (continueOnFail: true)
→ IF (check error)
├─ [Success Path]
└─ [Error Path] → Log → Retry or Alert
```
**IF condition**:
```javascript
{{$json.error}} is empty
// OR
{{$json.statusCode}} < 400
```
---
## Rate Limiting
### Pattern 1: Wait Between Requests
```
Split In Batches (1 item per batch)
→ HTTP Request
→ Wait (1 second)
→ Loop
```
### Pattern 2: Exponential Backoff
```javascript
// Code node
const maxRetries = 3;
let retryCount = $json.retryCount || 0;
if ($json.error && retryCount < maxRetries) {
const delay = Math.pow(2, retryCount) * 1000; // 1s, 2s, 4s
return [{
json: {
...$json,
retryCount: retryCount + 1,
waitTime: delay
}
}];
}
```
### Pattern 3: Respect Rate Limit Headers
```javascript
// Code node - check rate limit
const headers = $input.first().json.headers;
const remaining = parseInt(headers['x-ratelimit-remaining'] || '999');
const resetTime = parseInt(headers['x-ratelimit-reset'] || '0');
if (remaining < 10) {
const now = Math.floor(Date.now() / 1000);
const waitSeconds = resetTime - now;
return [{
json: {
shouldWait: true,
waitSeconds: Math.max(waitSeconds, 0)
}
}];
}
return [{ json: { shouldWait: false } }];
```
---
## Request Configuration
### GET Request
```javascript
{
method: "GET",
url: "https://api.example.com/users",
sendQuery: true,
queryParameters: {
"page": "1",
"limit": "100",
"filter": "active"
}
}
```
### POST Request (JSON Body)
```javascript
{
method: "POST",
url: "https://api.example.com/users",
sendBody: true,
bodyParametersJson: JSON.stringify({
name: "={{$json.name}}",
email: "={{$json.email}}",
role: "user"
})
}
```
### POST Request (Form Data)
```javascript
{
method: "POST",
url: "https://api.example.com/upload",
sendBody: true,
bodyParametersUi: {
parameter: [
{ name: "file", value: "={{$json.fileData}}" },
{ name: "filename", value: "={{$json.filename}}" }
]
},
sendHeaders: true,
headerParameters: {
"Content-Type": "multipart/form-data"
}
}
```
### PUT/PATCH Request (Update)
```javascript
{
method: "PATCH",
url: "https://api.example.com/users/={{$json.userId}}",
sendBody: true,
bodyParametersJson: JSON.stringify({
status: "active",
last_updated: "={{$now}}"
})
}
```
### DELETE Request
```javascript
{
method: "DELETE",
url: "https://api.example.com/users/={{$json.userId}}"
}
```
---
## Error Handling Patterns
### Pattern 1: Retry on Failure
```
HTTP Request (continueOnFail: true)
→ IF (error occurred)
└→ Wait (5 seconds)
└→ HTTP Request (retry)
```
### Pattern 2: Fallback API
```
HTTP Request (Primary API, continueOnFail: true)
→ IF (failed)
└→ HTTP Request (Fallback API)
```
### Pattern 3: Error Trigger Workflow
**Main Workflow**:
```
HTTP Request → Process Data
```
**Error Workflow**:
```
Error Trigger
→ Set (extract error details)
→ Slack (alert team)
→ Database (log error for analysis)
```
### Pattern 4: Circuit Breaker
```javascript
// Code node - circuit breaker logic
const failures = $json.recentFailures || 0;
const threshold = 5;
if (failures >= threshold) {
throw new Error('Circuit breaker open - too many failures');
}
return [{ json: { canProceed: true } }];
```
---
## Response Transformation
### Extract Nested Data
```javascript
// Code node
const response = $input.first().json;
return response.data.items.map(item => ({
json: {
id: item.id,
name: item.attributes.name,
email: item.attributes.contact.email
}
}));
```
### Flatten Arrays
```javascript
// Code node - flatten nested array
const items = $input.all();
const flattened = items.flatMap(item =>
item.json.results.map(result => ({
json: {
parent_id: item.json.id,
...result
}
}))
);
return flattened;
```
### Combine Multiple API Responses
```
HTTP Request 1 (users)
→ Set (store users)
→ HTTP Request 2 (orders for each user)
→ Merge (combine users + orders)
```
---
## Testing & Debugging
### 1. Test with Manual Trigger
Replace Schedule with Manual Trigger for testing
### 2. Use Postman/Insomnia First
- Test API outside n8n
- Understand response structure
- Verify authentication
### 3. Log Responses
```javascript
// Code node - log for debugging
console.log('API Response:', JSON.stringify($input.first().json, null, 2));
return $input.all();
```
### 4. Check Execution Data
- View node output in n8n UI
- Check headers, body, status code
- Verify data structure
### 5. Use Binary Data Properly
For file downloads:
```javascript
{
method: "GET",
url: "https://api.example.com/download/file.pdf",
responseFormat: "file", // Important for binary data
outputPropertyName: "data"
}
```
---
## Performance Optimization
### 1. Parallel Requests
Use **Split In Batches** with multiple items:
```
Set (create array of IDs)
→ Split In Batches (10 items per batch)
→ HTTP Request (processes all 10 in parallel)
→ Loop
```
### 2. Caching
```
IF (check cache exists)
├─ [Cache Hit] → Use cached data
└─ [Cache Miss] → HTTP Request → Store in cache
```
### 3. Conditional Fetching
Only fetch if data changed:
```
HTTP Request (GET with If-Modified-Since header)
→ IF (status === 304)
└─ Use existing data
→ IF (status === 200)
└─ Process new data
```
### 4. Batch API Calls
If API supports batch operations:
```javascript
{
method: "POST",
url: "https://api.example.com/batch",
bodyParametersJson: JSON.stringify({
requests: $json.items.map(item => ({
method: "GET",
url: `/users/${item.id}`
}))
})
}
```
---
## Common Gotchas
### 1. ❌ Wrong: Hardcoded URLs
```javascript
url: "https://api.example.com/prod/users"
```
### ✅ Correct: Use environment variables
```javascript
url: "={{$env.API_BASE_URL}}/users"
```
### 2. ❌ Wrong: Credentials in parameters
```javascript
headerParameters: {
"Authorization": "Bearer sk-abc123xyz" // ❌ Exposed!
}
```
### ✅ Correct: Use credentials system
```javascript
authentication: "predefinedCredentialType",
nodeCredentialType: "httpHeaderAuth"
```
### 3. ❌ Wrong: No error handling
```javascript
HTTP Request → Process (fails if API down)
```
### ✅ Correct: Handle errors
```javascript
HTTP Request (continueOnFail: true) → IF (error) → Handle
```
### 4. ❌ Wrong: Blocking on large responses
Processing 10,000 items synchronously
### ✅ Correct: Use batching
```
Split In Batches (100 items) → Process → Loop
```
---
## Real Template Examples
From n8n template library (892 API integration templates):
**GitHub to Notion**:
```
Schedule → HTTP Request (GitHub API) → Transform → HTTP Request (Notion API)
```
**Weather to Slack**:
```
Schedule → HTTP Request (Weather API) → Set (format) → Slack
```
**CRM Sync**:
```
Schedule → HTTP Request (CRM A) → Transform → HTTP Request (CRM B)
```
Use `search_templates({query: "http api"})` to find more!
---
## Checklist for API Integration
### Planning
- [ ] Test API with Postman/curl first
- [ ] Understand response structure
- [ ] Check rate limits
- [ ] Review authentication method
- [ ] Plan error handling
### Implementation
- [ ] Use credentials (never hardcode)
- [ ] Configure proper HTTP method
- [ ] Set correct headers (Content-Type, Accept)
- [ ] Handle pagination if needed
- [ ] Add query parameters properly
### Error Handling
- [ ] Set continueOnFail: true if needed
- [ ] Check response status codes
- [ ] Implement retry logic
- [ ] Add Error Trigger workflow
- [ ] Alert on failures
### Performance
- [ ] Use batching for large datasets
- [ ] Add rate limiting if needed
- [ ] Consider caching
- [ ] Test with production load
### Security
- [ ] Use HTTPS only
- [ ] Store secrets in credentials
- [ ] Validate API responses
- [ ] Use environment variables
---
## Summary
**Key Points**:
1. **Authentication** via credentials system (never hardcode)
2. **Error handling** is critical (continueOnFail + IF checks)
3. **Pagination** for large datasets
4. **Rate limiting** to respect API limits
5. **Transform responses** to match your needs
**Pattern**: Trigger → HTTP Request → Transform → Action → Error Handler
**Related**:
- [webhook_processing.md](webhook_processing.md) - Receiving HTTP requests
- [database_operations.md](database_operations.md) - Storing API data

773
.opencode/skills-off/n8n-workflow-patterns/scheduled_tasks.md Normal file
View File

@@ -0,0 +1,773 @@
# Scheduled Tasks Pattern
**Use Case**: Recurring automation workflows that run automatically on a schedule.
---
## Pattern Structure
```
Schedule Trigger → [Fetch Data] → [Process] → [Deliver] → [Log/Notify]
```
**Key Characteristic**: Time-based automated execution
---
## Core Components
### 1. Schedule Trigger
**Purpose**: Execute workflow at specified times
**Modes**:
- **Interval** - Every X minutes/hours/days
- **Cron** - Specific times (advanced)
- **Days & Hours** - Simple recurring schedule
### 2. Data Source
**Common sources**:
- HTTP Request (APIs)
- Database queries
- File reads
- Service-specific nodes
### 3. Processing
**Typical operations**:
- Filter/transform data
- Aggregate statistics
- Generate reports
- Check conditions
### 4. Delivery
**Output channels**:
- Email
- Slack/Discord/Teams
- File storage
- Database writes
### 5. Logging
**Purpose**: Track execution history
**Methods**:
- Database log entries
- File append
- Monitoring service
---
## Schedule Configuration
### Interval Mode
**Best for**: Simple recurring tasks
**Examples**:
```javascript
// Every 15 minutes
{
mode: "interval",
interval: 15,
unit: "minutes"
}
// Every 2 hours
{
mode: "interval",
interval: 2,
unit: "hours"
}
// Every day at midnight
{
mode: "interval",
interval: 1,
unit: "days"
}
```
### Days & Hours Mode
**Best for**: Specific days and times
**Examples**:
```javascript
// Weekdays at 9 AM
{
mode: "daysAndHours",
days: ["monday", "tuesday", "wednesday", "thursday", "friday"],
hour: 9,
minute: 0
}
// Every Monday at 6 PM
{
mode: "daysAndHours",
days: ["monday"],
hour: 18,
minute: 0
}
```
### Cron Mode (Advanced)
**Best for**: Complex schedules
**Examples**:
```javascript
// Every weekday at 9 AM
{
mode: "cron",
expression: "0 9 * * 1-5"
}
// First day of every month at midnight
{
mode: "cron",
expression: "0 0 1 * *"
}
// Every 15 minutes during business hours (9 AM - 5 PM) on weekdays
{
mode: "cron",
expression: "*/15 9-17 * * 1-5"
}
```
**Cron format**: `minute hour day month weekday`
- `*` = any value
- `*/15` = every 15 units
- `1-5` = range (Monday-Friday)
- `1,15` = specific values
**Cron examples**:
```
0 */6 * * * Every 6 hours
0 9,17 * * * At 9 AM and 5 PM daily
0 0 * * 0 Every Sunday at midnight
*/30 * * * * Every 30 minutes
0 0 1,15 * * 1st and 15th of each month
```
---
## Common Use Cases
### 1. Daily Reports
**Flow**: Schedule → Fetch data → Aggregate → Format → Email
**Example** (Sales report):
```
1. Schedule (daily at 9 AM)
2. Postgres (query yesterday's sales)
SELECT date, SUM(amount) as total, COUNT(*) as orders
FROM orders
WHERE date = CURRENT_DATE - INTERVAL '1 day'
GROUP BY date
3. Code (calculate metrics)
- Total revenue
- Order count
- Average order value
- Comparison to previous day
4. Set (format email body)
Subject: Daily Sales Report - {{$json.date}}
Body: Formatted HTML with metrics
5. Email (send to team@company.com)
6. Slack (post summary to #sales)
```
### 2. Data Synchronization
**Flow**: Schedule → Fetch from source → Transform → Write to target
**Example** (CRM to data warehouse sync):
```
1. Schedule (every hour)
2. Set (store last sync time)
SELECT MAX(synced_at) FROM sync_log
3. HTTP Request (fetch new CRM contacts since last sync)
GET /api/contacts?updated_since={{$json.last_sync}}
4. IF (check if new records exist)
5. Set (transform CRM schema to warehouse schema)
6. Postgres (warehouse - INSERT new contacts)
7. Postgres (UPDATE sync_log SET synced_at = NOW())
8. IF (error occurred)
└─ Slack (alert #data-team)
```
### 3. Monitoring & Health Checks
**Flow**: Schedule → Check endpoints → Alert if down
**Example** (Website uptime monitor):
```
1. Schedule (every 5 minutes)
2. HTTP Request (GET https://example.com/health)
- timeout: 10 seconds
- continueOnFail: true
3. IF (status !== 200 OR response_time > 2000ms)
4. Redis (check alert cooldown - don't spam)
- Key: alert:website_down
- TTL: 30 minutes
5. IF (no recent alert sent)
6. [Alert Actions]
├─ Slack (notify #ops-team)
├─ PagerDuty (create incident)
├─ Email (alert@company.com)
└─ Redis (set alert cooldown)
7. Postgres (log uptime check result)
```
### 4. Cleanup & Maintenance
**Flow**: Schedule → Find old data → Archive/Delete → Report
**Example** (Database cleanup):
```
1. Schedule (weekly on Sunday at 2 AM)
2. Postgres (find old records)
SELECT * FROM logs
WHERE created_at < NOW() - INTERVAL '90 days'
LIMIT 10000
3. IF (records exist)
4. Code (export to JSON for archive)
5. Google Drive (upload archive file)
- Filename: logs_archive_{{$now.format('YYYY-MM-DD')}}.json
6. Postgres (DELETE archived records)
DELETE FROM logs
WHERE id IN ({{$json.archived_ids}})
7. Slack (report: "Archived X records, deleted Y records")
```
### 5. Data Enrichment
**Flow**: Schedule → Find incomplete records → Enrich → Update
**Example** (Enrich contacts with company data):
```
1. Schedule (nightly at 3 AM)
2. Postgres (find contacts without company data)
SELECT id, email, domain FROM contacts
WHERE company_name IS NULL
AND created_at > NOW() - INTERVAL '7 days'
LIMIT 100
3. Split In Batches (10 contacts per batch)
4. HTTP Request (call Clearbit enrichment API)
- For each contact domain
- Rate limit: wait 1 second between batches
5. Set (map API response to database schema)
6. Postgres (UPDATE contacts with company data)
7. Wait (1 second - rate limiting)
8. Loop (back to step 4 until all batches processed)
9. Email (summary: "Enriched X contacts")
```
### 6. Backup Automation
**Flow**: Schedule → Export data → Compress → Store → Verify
**Example** (Database backup):
```
1. Schedule (daily at 2 AM)
2. Code (execute pg_dump)
const { exec } = require('child_process');
exec('pg_dump -h db.example.com mydb > backup.sql')
3. Code (compress backup)
const zlib = require('zlib');
// Compress backup.sql to backup.sql.gz
4. AWS S3 (upload compressed backup)
- Bucket: backups
- Key: db/backup-{{$now.format('YYYY-MM-DD')}}.sql.gz
5. AWS S3 (list old backups)
- Keep last 30 days only
6. AWS S3 (delete old backups)
7. IF (error occurred)
├─ PagerDuty (critical alert)
└─ Email (backup failed!)
ELSE
└─ Slack (#devops: "✅ Backup completed")
```
### 7. Content Publishing
**Flow**: Schedule → Fetch content → Format → Publish
**Example** (Automated social media posts):
```
1. Schedule (every 3 hours during business hours)
- Cron: 0 9,12,15,18 * * 1-5
2. Google Sheets (read content queue)
- Sheet: "Scheduled Posts"
- Filter: status=pending AND publish_time <= NOW()
3. IF (posts available)
4. HTTP Request (shorten URLs in post)
5. HTTP Request (POST to Twitter API)
6. HTTP Request (POST to LinkedIn API)
7. Google Sheets (update status=published)
8. Slack (notify #marketing: "Posted: {{$json.title}}")
```
---
## Timezone Considerations
### Set Workflow Timezone
```javascript
// In workflow settings
{
timezone: "America/New_York" // EST/EDT
}
```
### Common Timezones
```
America/New_York - Eastern (US)
America/Chicago - Central (US)
America/Denver - Mountain (US)
America/Los_Angeles - Pacific (US)
Europe/London - GMT/BST
Europe/Paris - CET/CEST
Asia/Tokyo - JST
Australia/Sydney - AEDT
UTC - Universal Time
```
### Handle Daylight Saving
**Best practice**: Use timezone-aware scheduling
```javascript
// ❌ Bad: UTC schedule for "9 AM local"
// Will be off by 1 hour during DST transitions
// ✅ Good: Set workflow timezone
{
timezone: "America/New_York",
schedule: {
mode: "daysAndHours",
hour: 9 // Always 9 AM Eastern, regardless of DST
}
}
```
---
## Error Handling
### Pattern 1: Error Trigger Workflow
**Main workflow**: Normal execution
**Error workflow**: Alerts and recovery
**Main**:
```
Schedule → Fetch → Process → Deliver
```
**Error**:
```
Error Trigger (for main workflow)
→ Set (extract error details)
→ Slack (#ops-team: "❌ Scheduled job failed")
→ Email (admin alert)
→ Postgres (log error for analysis)
```
### Pattern 2: Retry with Backoff
```
Schedule → HTTP Request (continueOnFail: true)
→ IF (error)
├─ Wait (5 minutes)
├─ HTTP Request (retry 1)
└─ IF (still error)
├─ Wait (15 minutes)
├─ HTTP Request (retry 2)
└─ IF (still error)
└─ Alert admin
```
### Pattern 3: Partial Failure Handling
```
Schedule → Split In Batches
→ Process (continueOnFail: true)
→ Code (track successes and failures)
→ Report:
"✅ Processed: 95/100"
"❌ Failed: 5/100"
```
---
## Performance Optimization
### 1. Batch Processing
For large datasets:
```
Schedule → Query (LIMIT 10000)
→ Split In Batches (100 items)
→ Process batch
→ Loop
```
### 2. Parallel Processing
When operations are independent:
```
Schedule
├─ [Branch 1: Update DB]
├─ [Branch 2: Send emails]
└─ [Branch 3: Generate report]
→ Merge (wait for all) → Final notification
```
### 3. Skip if Already Running
Prevent overlapping executions:
```
Schedule → Redis (check lock)
→ IF (lock exists)
└─ End (skip this execution)
→ ELSE
├─ Redis (set lock, TTL 30 min)
├─ [Execute workflow]
└─ Redis (delete lock)
```
### 4. Early Exit on No Data
Don't waste time if nothing to process:
```
Schedule → Query (check if work exists)
→ IF (no results)
└─ End workflow (exit early)
→ ELSE
└─ Process data
```
---
## Monitoring & Logging
### Pattern 1: Execution Log Table
```sql
CREATE TABLE workflow_executions (
id SERIAL PRIMARY KEY,
workflow_name VARCHAR(255),
started_at TIMESTAMP,
completed_at TIMESTAMP,
status VARCHAR(50),
records_processed INT,
error_message TEXT
);
```
**Log execution**:
```
Schedule
→ Set (record start)
→ [Workflow logic]
→ Postgres (INSERT execution log)
```
### Pattern 2: Metrics Collection
```
Schedule → [Execute]
→ Code (calculate metrics)
- Duration
- Records processed
- Success rate
→ HTTP Request (send to monitoring system)
- Datadog, Prometheus, etc.
```
### Pattern 3: Summary Notifications
Daily/weekly execution summaries:
```
Schedule (daily at 6 PM) → Query execution logs
→ Code (aggregate today's executions)
→ Email (summary report)
"Today's Workflow Executions:
- 24/24 successful
- 0 failures
- Avg duration: 2.3 min"
```
---
## Testing Scheduled Workflows
### 1. Use Manual Trigger for Testing
**Development pattern**:
```
Manual Trigger (for testing)
→ [Same workflow logic]
→ [Outputs]
// Once tested, replace with Schedule Trigger
```
### 2. Test with Different Times
```javascript
// Code node - simulate different times
const testTime = new Date('2024-01-15T09:00:00Z');
return [{ json: { currentTime: testTime } }];
```
### 3. Dry Run Mode
```
Schedule → Set (dryRun: true)
→ IF (dryRun)
└─ Log what would happen (don't execute)
→ ELSE
└─ Execute normally
```
### 4. Shorter Interval for Testing
```javascript
// Testing: every 1 minute
{
mode: "interval",
interval: 1,
unit: "minutes"
}
// Production: every 1 hour
{
mode: "interval",
interval: 1,
unit: "hours"
}
```
---
## Common Gotchas
### 1. ❌ Wrong: Ignoring timezone
```javascript
Schedule (9 AM) // 9 AM in which timezone?
```
### ✅ Correct: Set workflow timezone
```javascript
// Workflow settings
{
timezone: "America/New_York"
}
```
### 2. ❌ Wrong: Overlapping executions
```
Schedule (every 5 min) → Long-running task (10 min)
// Two executions running simultaneously!
```
### ✅ Correct: Add execution lock
```
Schedule → Redis (check lock)
→ IF (locked) → Skip
→ ELSE → Execute
```
### 3. ❌ Wrong: No error handling
```
Schedule → API call → Process (fails silently)
```
### ✅ Correct: Add error workflow
```
Main: Schedule → Execute
Error: Error Trigger → Alert
```
### 4. ❌ Wrong: Processing all data at once
```
Schedule → SELECT 1000000 records → Process (OOM)
```
### ✅ Correct: Batch processing
```
Schedule → SELECT with pagination → Split In Batches → Process
```
### 5. ❌ Wrong: Hardcoded dates
```javascript
query: "SELECT * FROM orders WHERE date = '2024-01-15'"
```
### ✅ Correct: Dynamic dates
```javascript
query: "SELECT * FROM orders WHERE date = CURRENT_DATE - INTERVAL '1 day'"
```
---
## Real Template Examples
From n8n template library:
**Template #2947** (Weather to Slack):
```
Schedule (daily 8 AM)
→ HTTP Request (weather API)
→ Set (format message)
→ Slack (post to #general)
```
**Daily backup**:
```
Schedule (nightly 2 AM)
→ Postgres (export data)
→ Google Drive (upload)
→ Email (confirmation)
```
**Monitoring**:
```
Schedule (every 5 min)
→ HTTP Request (health check)
→ IF (down) → PagerDuty alert
```
Use `search_templates({query: "schedule"})` to find more!
---
## Checklist for Scheduled Workflows
### Planning
- [ ] Define schedule frequency (interval, cron, days & hours)
- [ ] Set workflow timezone
- [ ] Estimate execution duration
- [ ] Plan for failures and retries
- [ ] Consider timezone and DST
### Implementation
- [ ] Configure Schedule Trigger
- [ ] Set workflow timezone in settings
- [ ] Add early exit for no-op cases
- [ ] Implement batch processing for large data
- [ ] Add execution logging
### Error Handling
- [ ] Create Error Trigger workflow
- [ ] Implement retry logic
- [ ] Add alert notifications
- [ ] Log errors for analysis
- [ ] Handle partial failures gracefully
### Monitoring
- [ ] Log each execution (start, end, status)
- [ ] Track metrics (duration, records, success rate)
- [ ] Set up daily/weekly summaries
- [ ] Alert on consecutive failures
- [ ] Monitor resource usage
### Testing
- [ ] Test with Manual Trigger first
- [ ] Verify timezone behavior
- [ ] Test error scenarios
- [ ] Check for overlapping executions
- [ ] Validate output quality
### Deployment
- [ ] Document workflow purpose
- [ ] Set up monitoring
- [ ] Configure alerts
- [ ] Activate workflow in n8n UI ⚠️ **Manual activation required** (API/MCP cannot activate)
- [ ] Test in production (short interval first)
- [ ] Monitor first few executions
---
## Advanced Patterns
### Dynamic Scheduling
**Change schedule based on conditions**:
```
Schedule (check every hour) → Code (check if it's time to run)
→ IF (business hours AND weekday)
└─ Execute workflow
→ ELSE
└─ Skip
```
### Dependent Schedules
**Chain workflows**:
```
Workflow A (daily 2 AM): Data sync
→ On completion → Trigger Workflow B
Workflow B: Generate report (depends on fresh data)
```
### Conditional Execution
**Skip based on external factors**:
```
Schedule → HTTP Request (check feature flag)
→ IF (feature enabled)
└─ Execute
→ ELSE
└─ Skip
```
---
## Summary
**Key Points**:
1. **Set workflow timezone** explicitly
2. **Batch processing** for large datasets
3. **Error handling** is critical (Error Trigger + retries)
4. **Prevent overlaps** with execution locks
5. **Monitor and log** all executions
**Pattern**: Schedule → Fetch → Process → Deliver → Log
**Schedule Modes**:
- **Interval**: Simple recurring (every X minutes/hours)
- **Days & Hours**: Specific days and times
- **Cron**: Advanced complex schedules
**Related**:
- [http_api_integration.md](http_api_integration.md) - Fetching data on schedule
- [database_operations.md](database_operations.md) - Scheduled database tasks
- [webhook_processing.md](webhook_processing.md) - Alternative to scheduling

545
.opencode/skills-off/n8n-workflow-patterns/webhook_processing.md Normal file
View File

@@ -0,0 +1,545 @@
# Webhook Processing Pattern
**Use Case**: Receive HTTP requests from external systems and process them instantly.
---
## Pattern Structure
```
Webhook → [Validate] → [Transform] → [Action] → [Response/Notify]
```
**Key Characteristic**: Instant event-driven processing
---
## Core Components
### 1. Webhook Node (Trigger)
**Purpose**: Create HTTP endpoint to receive data
**Configuration**:
```javascript
{
path: "form-submit", // URL path: https://n8n.example.com/webhook/form-submit
httpMethod: "POST", // GET, POST, PUT, DELETE
responseMode: "onReceived", // or "lastNode" for custom response
responseData: "allEntries" // or "firstEntryJson"
}
```
**Critical Gotcha**: Data is nested under `$json.body`
```javascript
❌ {{$json.email}}
✅ {{$json.body.email}}
```
### 2. Validation (Optional but Recommended)
**Purpose**: Verify incoming data before processing
**Options**:
- **IF node** - Check required fields exist
- **Code node** - Custom validation logic
- **Stop and Error** - Fail gracefully with message
**Example**:
```javascript
// IF node condition
{{$json.body.email}} is not empty AND
{{$json.body.name}} is not empty
```
### 3. Transformation
**Purpose**: Map webhook data to desired format
**Typical nodes**:
- **Set** - Field mapping
- **Code** - Complex transformations
**Example** (Set node):
```javascript
{
"user_email": "={{$json.body.email}}",
"user_name": "={{$json.body.name}}",
"timestamp": "={{$now}}"
}
```
### 4. Action
**Purpose**: Do something with the data
**Common actions**:
- Store in database (Postgres, MySQL, MongoDB)
- Send notification (Slack, Email, Discord)
- Call another API (HTTP Request)
- Update external system (CRM, support ticket)
### 5. Response (If responseMode: "lastNode")
**Purpose**: Send custom HTTP response
**Webhook Response Node**:
```javascript
{
statusCode: 200,
headers: {
"Content-Type": "application/json"
},
body: {
"status": "success",
"message": "Form received"
}
}
```
---
## Common Use Cases
### 1. Form Submissions
**Flow**: Form → Webhook → Validate → Database → Email Confirmation
**Example**:
```
1. Webhook (path: "contact-form", POST)
2. IF (check email & message not empty)
3. Postgres (insert into contacts table)
4. Email (send confirmation to user)
5. Slack (notify team in #leads)
6. Webhook Response ({"status": "success"})
```
**Real Data Access**:
```javascript
Name: {{$json.body.name}}
Email: {{$json.body.email}}
Message: {{$json.body.message}}
```
### 2. Payment Webhooks (Stripe, PayPal)
**Flow**: Payment Provider → Webhook → Verify → Update Database → Send Receipt
**Security**: Verify webhook signatures
```javascript
// Code node - verify Stripe signature
const crypto = require('crypto');
const signature = $input.item.headers['stripe-signature'];
const secret = $credentials.stripeWebhookSecret;
// Verify signature matches
const expectedSig = crypto
.createHmac('sha256', secret)
.update($input.item.body)
.digest('hex');
if (signature !== expectedSig) {
throw new Error('Invalid webhook signature');
}
return $input.item.body; // Return validated body
```
### 3. Chat Platform Integrations (Slack, Discord, Teams)
**Flow**: Chat Command → Webhook → Process → Respond
**Example** (Slack slash command):
```
1. Webhook (path: "slack-command", POST)
2. Code (parse Slack payload: $json.body.text, $json.body.user_id)
3. HTTP Request (fetch data from API)
4. Set (format Slack message)
5. Webhook Response (immediate Slack response)
```
**Slack Data Access**:
```javascript
Command: {{$json.body.command}}
Text: {{$json.body.text}}
User ID: {{$json.body.user_id}}
Channel ID: {{$json.body.channel_id}}
```
### 4. GitHub/GitLab Webhooks
**Flow**: Git Event → Webhook → Parse → Notify/Deploy
**Example** (new PR notification):
```
1. Webhook (path: "github", POST)
2. IF (check $json.body.action equals "opened")
3. Set (extract PR details: title, author, url)
4. Slack (notify #dev-team)
5. Webhook Response (200 OK)
```
**GitHub Data Access**:
```javascript
Event Type: {{$json.headers['x-github-event']}}
Action: {{$json.body.action}}
PR Title: {{$json.body.pull_request.title}}
Author: {{$json.body.pull_request.user.login}}
URL: {{$json.body.pull_request.html_url}}
```
### 5. IoT Device Data
**Flow**: Device → Webhook → Validate → Store → Alert (if threshold)
**Example** (temperature sensor):
```
1. Webhook (path: "sensor-data", POST)
2. Set (extract sensor readings)
3. Postgres (insert into sensor_readings)
4. IF (temperature > 80)
5. Email (alert admin)
```
---
## Webhook Data Structure
### Standard Structure
```json
{
"headers": {
"content-type": "application/json",
"user-agent": "...",
"x-custom-header": "..."
},
"params": {
"id": "123" // From URL: /webhook/form/:id
},
"query": {
"token": "abc" // From URL: /webhook/form?token=abc
},
"body": {
// ⚠️ YOUR DATA IS HERE!
"name": "John",
"email": "john@example.com"
}
}
```
### Accessing Different Parts
```javascript
// Headers
{{$json.headers['content-type']}}
{{$json.headers['x-api-key']}}
// URL Parameters
{{$json.params.id}}
// Query Parameters
{{$json.query.token}}
{{$json.query.page}}
// Body (MOST COMMON)
{{$json.body.email}}
{{$json.body.user.name}}
{{$json.body.items[0].price}}
```
---
## Authentication & Security
### 1. Query Parameter Token
**Simple but less secure**
```javascript
// IF node - validate token
{{$json.query.token}} equals "your-secret-token"
```
### 2. Header-Based Auth
**Better security**
```javascript
// IF node - check header
{{$json.headers['x-api-key']}} equals "your-api-key"
```
### 3. Signature Verification
**Best security** (for webhooks from services like Stripe, GitHub)
```javascript
// Code node
const crypto = require('crypto');
const signature = $input.item.headers['x-signature'];
const secret = $credentials.webhookSecret;
const calculatedSig = crypto
.createHmac('sha256', secret)
.update(JSON.stringify($input.item.body))
.digest('hex');
if (signature !== `sha256=${calculatedSig}`) {
throw new Error('Invalid signature');
}
return $input.item.body;
```
### 4. IP Whitelist
**Restrict access by IP** (n8n workflow settings)
- Configure in workflow settings
- Only allow specific IP ranges
- Use for internal systems
---
## Response Modes
### onReceived (Default)
**Behavior**: Immediate 200 OK response, workflow continues in background
**Use when**:
- Long-running workflows
- Response doesn't depend on workflow result
- Fire-and-forget processing
**Configuration**:
```javascript
{
responseMode: "onReceived",
responseCode: 200
}
```
### lastNode (Custom Response)
**Behavior**: Wait for workflow completion, send custom response
**Use when**:
- Need to return data to caller
- Synchronous processing required
- Form submissions with confirmation
**Configuration**:
```javascript
{
responseMode: "lastNode"
}
```
**Then add Webhook Response node**:
```javascript
{
statusCode: 200,
headers: {
"Content-Type": "application/json"
},
body: {
"id": "={{$json.record_id}}",
"status": "success"
}
}
```
---
## Error Handling
### Pattern 1: Try-Catch with Error Trigger
```
Main Flow:
Webhook → [nodes...] → Success Response
Error Flow:
Error Trigger → Log Error → Slack Alert → Error Response
```
**Error Trigger Configuration**:
```javascript
{
workflowId: "current-workflow-id"
}
```
**Error Response** (if responseMode: "lastNode"):
```javascript
{
statusCode: 500,
body: {
"status": "error",
"message": "Processing failed"
}
}
```
### Pattern 2: Validation Early Exit
```
Webhook → IF (validate) → [True: Process]
└→ [False: Error Response]
```
**False Branch Response**:
```javascript
{
statusCode: 400,
body: {
"status": "error",
"message": "Invalid data: missing email"
}
}
```
### Pattern 3: Continue On Fail
**Per-node setting**: Continue even if node fails
**Use case**: Non-critical notifications
```
Webhook → Database (critical) → Slack (continueOnFail: true)
```
---
## Testing Webhooks
### 1. Use Manual Trigger
Replace Webhook with Manual Trigger for testing:
```
Manual Trigger → [set test data] → rest of workflow
```
### 2. Use curl
```bash
curl -X POST https://n8n.example.com/webhook/form-submit \
-H "Content-Type: application/json" \
-d '{"email": "test@example.com", "name": "Test User"}'
```
### 3. Use Postman/Insomnia
- Create request collection
- Test different payloads
- Verify responses
### 4. Webhook.site
- Use webhook.site for testing
- Copy webhook.site URL to your service
- View requests and debug
---
## Performance Considerations
### Large Payloads
- Webhook timeout: 120 seconds (default)
- For large data, consider async processing:
```
Webhook → Queue (Redis/DB) → Response (immediate)
Separate Workflow:
Schedule → Check Queue → Process
```
### High Volume
- Use "Execute Once" mode if processing all items together
- Consider rate limiting
- Monitor execution times
- Scale n8n instance if needed
### Retries
- Webhook calls typically don't retry automatically
- Implement retry logic on caller side
- Or use queue pattern for guaranteed processing
---
## Common Gotchas
### 1. ❌ Wrong: Accessing webhook data
```javascript
{{$json.email}} // Empty or undefined
```
### ✅ Correct
```javascript
{{$json.body.email}} // Data is under .body
```
### 2. ❌ Wrong: Response mode confusion
Using Webhook Response node with responseMode: "onReceived" (ignored)
### ✅ Correct
Set responseMode: "lastNode" to use Webhook Response node
### 3. ❌ Wrong: No validation
Assuming data is always present and valid
### ✅ Correct
Validate data early with IF node or Code node
### 4. ❌ Wrong: Hardcoded paths
Using same path for dev/prod
### ✅ Correct
Use environment variables: `{{$env.WEBHOOK_PATH_PREFIX}}/form-submit`
---
## Real Template Examples
From n8n template library (1,085 webhook templates):
**Simple Form to Slack**:
```
Webhook → Set → Slack
```
**Payment Processing**:
```
Webhook → Verify Signature → Update Database → Send Receipt → Notify Admin
```
**Chat Bot**:
```
Webhook → Parse Command → AI Agent → Format Response → Webhook Response
```
Use `search_templates({query: "webhook"})` to find more!
---
## Checklist for Webhook Workflows
### Setup
- [ ] Choose descriptive webhook path
- [ ] Configure HTTP method (POST most common)
- [ ] Choose response mode (onReceived vs lastNode)
- [ ] Test webhook URL before connecting services
### Security
- [ ] Add authentication (token, signature, IP whitelist)
- [ ] Validate incoming data
- [ ] Sanitize user input (if storing/displaying)
- [ ] Use HTTPS (always)
### Data Handling
- [ ] Remember data is under $json.body
- [ ] Handle missing fields gracefully
- [ ] Transform data to desired format
- [ ] Log important data (for debugging)
### Error Handling
- [ ] Add Error Trigger workflow
- [ ] Validate required fields
- [ ] Return appropriate error responses
- [ ] Alert team on failures
### Testing
- [ ] Test with curl/Postman
- [ ] Test error scenarios
- [ ] Verify response format
- [ ] Monitor first executions
---
## Summary
**Key Points**:
1. **Data under $json.body** (most common mistake!)
2. **Validate early** to catch bad data
3. **Choose response mode** based on use case
4. **Secure webhooks** with auth
5. **Handle errors** gracefully
**Pattern**: Webhook → Validate → Transform → Action → Response
**Related**:
- [n8n Expression Syntax](../../n8n-expression-syntax/SKILL.md) - Accessing webhook data correctly
- [http_api_integration.md](http_api_integration.md) - Making HTTP requests in response