Data Download Tasks

Overview

Download tasks export historical factor data into files stored in S3 (or compatible object storage). Each task belongs to a project but is managed through a dedicated endpoint namespace.

• Base Path: /api/v2/projects/{project_id}/data-download-tasks/
• Caching: Task listings are cached briefly per organization.

Endpoint Summary

Method	Path	Purpose	Notes
GET	/projects/{project_id}/data-download-tasks/	List download tasks for the organization	See List Tasks.
POST	/projects/{project_id}/data-download-tasks/	Create a new download task	See Create Task.
DELETE	/projects/{project_id}/data-download-tasks/{task_id}/	Delete a task	See Delete Task.
GET	/projects/{project_id}/data-download-tasks/{task_id}/download/	Retrieve pre-signed file URL	See Retrieve Download URL.

Update and partial update operations return HTTP 405; download tasks are immutable except for deletion.

List Tasks

GET /api/v2/projects/{project_id}/data-download-tasks/
Authorization: Bearer <token>

Response (example):

[
  {
    "id": 15,
    "name": "ColdRoom_3factors_20240201-20240203",
    "device_id": 128,
    "factors": ["d-1000-abcd-01", "d-1000-abcd-02"],
    "start_time": 1706745600,
    "end_time": 1706918400,
    "format": "csv",
    "status": "pending",
    "file_path": "128/ColdRoom_3factors_20240201-20240203_ab12cd34.csv",
    "create_time": "2024-02-01T08:00:00Z"
  }
]

Create Task

POST /api/v2/projects/{project_id}/data-download-tasks/
Authorization: Bearer <token>
Content-Type: application/json

{
  "device_id": 128,
  "factors": ["d-1000-abcd-01"],
  "start_time": 1706745600,
  "end_time": 1706918400,
  "format": "csv",
  "name": "Optional custom name"
}

Validation Rules

1. device_id must exist and belong to the organization.
2. start_time (and optional end_time) must be Unix timestamps (seconds).
3. factors should contain full agri_id values; used to build the historical query.
4. format defaults to csv if omitted.
5. If name is missing, the backend generates one using device name, factor count, and time range.

Backend Workflow

• Builds the time range filter for the historical store.
• Generates a safe filename plus UUID, stored in file_path.
• Persists the task with status pending.
• Publishes an export job message so the async worker can collect data and generate the file.
• Sets status=pending; the worker updates status to completed or failed externally.

Retrieve Download URL

GET /api/v2/projects/{project_id}/data-download-tasks/{task_id}/download/
Authorization: Bearer <token>

• Requires status=completed; otherwise returns HTTP 400 with {"error": "文件尚未生成完成"}.
• Generates a pre-signed S3 URL valid for 24 hours.
• Verifies the object exists via head_object; returns 404 if missing.

Response:

{
  "download_url": "https://bucket.s3.region.amazonaws.com/128/...&X-Amz-SignedHeaders=host"
}

Delete Task

DELETE /api/v2/projects/{project_id}/data-download-tasks/{task_id}/
Authorization: Bearer <token>

Deletes the task record. The exported file is not automatically removed from object storage; manage lifecycle policies separately.

Error Codes

Status	Code	Description
400	INVALID_TIME_RANGE | MISSING_DEVICE_ID	Start/end time invalid or required fields missing
401	UNAUTHORIZED	Missing or invalid token
403	FORBIDDEN	Caller lacks operator/manager role
404	DEVICE_NOT_FOUND | DOWNLOAD_TASK_NOT_FOUND	Device or task not accessible
405	METHOD_NOT_ALLOWED	Attempted to update a task
500	DOWNLOAD_LINK_FAILURE	Failed to generate pre-signed URL

Operational Notes

1. Deduplication: The export queue uses the task ID to prevent duplicate jobs.
2. Formats: Currently supports CSV; extend format handling when adding new export formats.
3. Field Mapping: The backend enriches export requests with field names so the resulting file has readable headers.
4. Lifecycle: Implement S3 lifecycle rules to expire export files after a retention period if they are large or sensitive.
5. Status Polling: Since exports are asynchronous, poll the list endpoint or watch status fields to know when to download.