When to use idempotency for a publishing API

Once a personal Flask blog has an automated publishing API, the next operational problem is rarely the first successful post. The risk is the second request: a retry after a timeout, a manual rerun by an operator, a Chinese article that succeeds while the English article fails, or a search submission that points to a URL created by the wrong attempt. For a small site such as Stepnex, especially during an ad-network review sprint, publishing needs to be boring and traceable. The site should add high-quality technical articles without creating duplicate slugs, broken translation pairs, half-written bodies, or search submissions for pages that do not exist.

Idempotency means that the same publishing intent can be executed more than once and still produce the same durable result. It does not mean ignoring errors. It means that a retry with the same request fingerprint returns the original article instead of creating a second copy. This pattern is useful for JSON-based article generation, scheduled publishing, draft promotion, bilingual publishing, and post-publish search submission. It builds on the existing Stepnex article about making a Flask auto-publishing API more reliable, but focuses on duplicate prevention, conflict responses, audit records, and rollback drills.

Design choices: constrain the entry point first

Keep the first version small. A personal blog does not need a queue, an event bus, and a separate workflow engine before the basic write path is correct. Start with three layers. The first layer is the request contract: slug, language, translation_key, and either Idempotency-Key or a deterministic content hash must be stable. The second layer is the database constraint: article.slug must be unique, and the audit table must have a unique idempotency key. The third layer is an audit record that captures the request hash, operator, status, article id, public URL, error message, and search submission status.

The key point is that the database must participate. A Python-level “check then insert” is not enough when two retries arrive at nearly the same time. The code can make the response friendly, but the database should be the final guardrail against duplicate slugs and duplicate idempotency keys.

Step 1: create a stable request fingerprint

If the client can send an Idempotency-Key header, use it. If not, derive one from fields that define the publishing intent. Do not include timestamps, random suffixes, or environment-specific URLs in the fingerprint, because they change between retries.

import hashlib
import json

def payload_hash(payload: dict) -> str:
    body = json.dumps(payload, ensure_ascii=False, sort_keys=True)
    return hashlib.sha256(body.encode("utf-8")).hexdigest()

def article_fingerprint(payload: dict) -> str:
    raw = "|".join([
        payload.get("language", ""),
        payload.get("slug", ""),
        payload.get("translation_key", ""),
        hashlib.sha256(payload.get("content_md", "").encode("utf-8")).hexdigest(),
    ])
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()

Add database constraints before changing the route. SQLite can enforce the same shape used later in MySQL or PostgreSQL:

create unique index if not exists uq_article_slug on article(slug);

create table if not exists publish_audit (
    id integer primary key autoincrement,
    idempotency_key text not null unique,
    article_slug text not null,
    language text not null,
    payload_hash text not null,
    status text not null,
    article_id integer,
    public_url text,
    error_message text,
    created_at text not null
);

Validate the configuration with a direct database check. Publish the same payload twice in a local or staging database. The first request should create the article and a success audit row. The second request should return the same article id and URL without adding another article.

sqlite3 database.db "select slug,count(*) from article group by slug having count(*)>1;"
sqlite3 database.db "select idempotency_key,status,article_id from publish_audit order by id desc limit 5;"

Step 2: return different responses for replay and conflict

The route should not collapse every problem into HTTP 500. Invalid payloads should return 400. Authentication failures should return 401 or 403. A true idempotent replay should return 200 with a flag such as idempotent_replay=true. A slug conflict should return 409 when an article with the same slug exists but the payload hash, language, or translation key does not match.

That distinction matters for automation. A timeout or 503 may be retried. A 409 should usually stop the job and ask for a human decision. A 401 means credentials are wrong and retrying will not help. A replay is not an error at all; it is the expected behavior after a safe retry.

@app.post("/api/articles/publish")
def publish_article():
    payload = request.get_json(silent=False)
    key = request.headers.get("Idempotency-Key") or article_fingerprint(payload)
    digest = payload_hash(payload)

    audit = PublishAudit.query.filter_by(idempotency_key=key).first()
    if audit and audit.status == "success":
        article = Article.query.get(audit.article_id)
        return jsonify({
            "ok": True,
            "id": article.id,
            "slug": article.slug,
            "language": article.language,
            "idempotent_replay": True,
        })

    existing = Article.query.filter_by(slug=payload["slug"]).first()
    if existing and getattr(existing, "content_hash", "") != digest:
        return jsonify({"ok": False, "error": "slug_conflict"}), 409

    article = create_article_from_payload(payload)
    db.session.add(PublishAudit(
        idempotency_key=key,
        article_slug=article.slug,
        language=article.language,
        payload_hash=digest,
        status="success",
        article_id=article.id,
    ))
    db.session.commit()
    return jsonify({"ok": True, "id": article.id, "slug": article.slug})

The pitfall is assuming that the pre-insert lookup is enough. It is not. Wrap the commit and catch the database integrity error. If the unique idempotency key already exists, read the audit row and return the previous result. If the slug already exists with a different hash, return 409. Log the path as created, replay, conflict, or failed so that the next review can tell what happened.

These steps also need a stable response schema. A publishing script should be able to parse the same fields for a new article and a replayed article: ok, id, slug, language, url, and idempotent_replay. Error responses should include a machine-readable value such as bad_payload, unauthorized, slug_conflict, or storage_failed. Avoid returning only a human sentence. During an incident, the operator should be able to take a slug from the failed job, search the application log, find the audit row, and compare it with the Nginx access log. When those three records agree, the publishing chain is reviewable.

Step 3: keep search submission compensatable

Search submission should happen after the article is committed and the public URL is known. Do not keep the database transaction open while waiting for a search platform. If Baidu submission fails, the article should remain published and the audit row should mark baidu_pending. A separate command can resubmit pending URLs later.

def after_publish(article_url: str, audit_id: int) -> None:
    try:
        result = submit_baidu_url(article_url)
    except Exception as exc:
        mark_audit(audit_id, "baidu_pending", str(exc)[:500])
    else:
        mark_audit(audit_id, "baidu_submitted", json.dumps(result, ensure_ascii=False))

Validate the public path, sitemap inclusion, and sprint script output:

curl -I https://stepnex.cn/en/article/flask-publish-api-idempotency-audit-en
curl -s https://stepnex.cn/sitemap.xml | grep flask-publish-api-idempotency-audit
python scripts/baidu_union_sprint.py --submit-limit 10

Missing credentials should be reported plainly. If BAIDU_PUSH_URL, BAIDU_SITE, or BAIDU_TOKEN is absent, the job should say so instead of pretending the URL was submitted. The same rule applies to Google Search Console configuration. A sitemap URL is not the same as confirmed indexing.

The compensation job should also have a small scope. For example, take the latest ten baidu_pending audit rows or the latest ten sitemap article URLs, submit them, and write back the response summary. Do not submit the full history on every run. Do not retry a permanent 4xx response forever. Treat credential errors, malformed site values, and URL encoding problems as configuration issues; treat network timeouts and temporary 5xx responses as retryable. This keeps the search submission stage useful without turning it into a noisy background process.

Step 4: make rollback an audited state change

Rollback should not start with deleting rows. A safer default is to move the article to draft or archived, record a rollback event, refresh sitemap or cache state, and keep enough evidence to explain what happened. This matters when a post was already submitted to search engines or linked from another article.

python scripts/publish_rollback.py   --slug flask-publish-api-idempotency-audit-en   --status draft   --reason "validation failed after publish"

The rollback script should update the article status, write an audit event, and trigger sitemap or cache refresh. Validate the article URL, category page, sitemap, and admin listing. A common mistake is changing the database row but leaving the URL in a cached sitemap. Another mistake is deleting the article while keeping audit records that point to a missing id.

Rollback validation should use the same steps every time. First, check the admin record and confirm the status changed. Second, request the public URL and confirm the expected behavior: visible, draft-only, archived, 404, or 410. Third, fetch the sitemap and category page to make sure the article appears or disappears consistently. Fourth, inspect the audit table and confirm the rollback reason is present. Fifth, keep a short operations note with the command, timestamp, and operator. The note does not need to be long, but it must be enough for the next person to understand why the article changed state.

Validation checklist

  • Publish the same JSON payload twice and confirm only one article row exists.
  • Send the same slug with a different body and confirm the API returns 409.
  • Confirm a replay returns the original article id, slug, language, and public URL.
  • Confirm Chinese and English articles use different slugs but share the same translation key.
  • Confirm 400, 401, 403, 409, and 5xx responses are logged with different labels.
  • Confirm Baidu submission failure leaves the article published and marks the audit row as pending.
  • Confirm sitemap and category pages reflect the final article status after rollback.

Pitfalls to avoid

Do not automatically append a random suffix to resolve a slug conflict. That makes the automation look successful while silently changing the public URL. Do not reuse the same idempotency key for a content revision; a revision should use an update endpoint or a new audit event. Do not save full tokens, cookies, or secrets in publish logs. Store request hashes, operator ids, status labels, and article ids instead.

Also avoid retrying every error. Network errors and 5xx responses can use a small retry budget. Payload errors, authentication failures, and conflicts need correction. This is the difference between a resilient publishing system and a script that repeats a bad request until the content library is messy.

Weekly review

Once a week, run a short review of the publishing chain. Check for duplicate slugs, failed audit rows, pending search submissions, broken translation pairs, missing sitemap entries, and rollback events without validation notes. Keep the review small enough to finish in ten minutes. The goal is not bureaucracy; the goal is to know whether generation, publishing, public access, sitemap, and search submission are still aligned. During an ad-network sprint, that kind of operational consistency is more valuable than pushing one more article with an unclear state.