Performance Observations &
Execution Characteristics
The DataForge engine has been validated across local and cloud execution environments using real-world datasets and production-representative configurations. All results reflect sustained throughput under controlled conditions and are reproducible with comparable inputs.
Performance Summary
Validated results across execution environments and target systems.
Verified Benchmark Record
Sustained end-to-end ingestion over the full dataset under warmed system conditions.
- Local PostgreSQL ingestion
- API execution path
- Sustained run over full dataset — no synthetic batching
- Executed under warmed system conditions
The observed improvement is consistent with enhanced write-ahead logging (WAL) efficiency and improved buffer pool utilization due to warmed execution state. These factors influence the destination system behavior, not the DataForge engine itself.
This benchmark reflects sustained, end-to-end ingestion performance under realistic conditions and is representative of achievable throughput on properly configured systems.
Key Observations
The System Is Not Compute-Bound
Across all environments, high throughput is sustained with CPU utilization remaining low relative to output and disk I/O pressure remaining minimal under streaming conditions.
Execution Model Determines Realized Throughput
Cloud results demonstrate a significant divergence between execution modes against identical infrastructure:
Destination System Characteristics Define Upper Bounds
The engine's performance ceiling is determined by the characteristics of the target system, not the ingestion pipeline itself.
- PostgreSQL supports 2.5M+ rows/sec under warmed, optimized conditions via COPY FROM STDIN — single streaming protocol call per batch
- SQL Server throughput is constrained by TDS bulk copy protocol overhead and UTF-16 encoding at the driver layer
Implementation Layer Impacts Throughput Efficiency
Observed differences between implementations on SQL Server under equivalent conditions:
Local vs Cloud Performance Characteristics
Local execution provides minimal network overhead, direct I/O access, and no container lifecycle constraints — producing the highest achievable throughput ceiling.
Cloud execution introduces network transfer overhead and service model constraints. Despite these differences, properly configured cloud execution remains within the same order of magnitude as local performance.
Concurrency and Scaling Behavior
Performance improves with concurrency: overhead is amortized across workloads, throughput increases without proportional infrastructure expansion, and per-unit cost decreases as utilization rises.
- Each Cloud Run Job handles one file at full throughput
- The concurrency ladder scales linearly against the storage write ceiling
- Infrastructure footprint remains stable as concurrency increases
A different approach to throughput.
The distinction is architectural, not incremental.
Increase throughput by adding infrastructure — more nodes, more services, more coordination layers. Each addition introduces latency, failure surface, and operational cost. The infrastructure is the answer to every throughput question.
Increases throughput by increasing concurrency within a stable execution boundary — amortizing overhead, not multiplying it. The infrastructure is fixed; the throughput scales with utilization.
Operational Implications
Use execution models that allow full CPU access for the duration of the job. Jobs over Services. Bare-metal over throttled containers. The mode is as important as the hardware.
Network overhead is real and measurable. Colocate the execution environment with the target system where possible. Same-region deployments confirmed no throughput difference between public IP and VPC private IP.
PostgreSQL and SQL Server carry different throughput ceilings under the same workload. System selection is a performance decision. The 8× gap exists at the protocol layer before DataForge is in the picture.
Language and driver selection affect how efficiently the engine's output is expressed. Native protocol integration outperforms abstraction layers at scale. The engine is constant; the delivery mechanism is a variable.
All results:
- Derived from real-world datasets (CourtListener public corpus)
- Executed under observable, repeatable conditions
- Reproducible with comparable configurations and hardware class
DataForge consistently delivers high-throughput data movement across environments, with performance bounded primarily by external system constraints rather than internal processing limits.
talk throughput?
Pilot discussions, investor conversations, enterprise architecture review, or technical deep-dives.