@hackage / shibuya-core

Supervised queue processing framework for Haskell

Latest0.8.0.1

Changelog

Changelog

0.8.0.1 — 2026-07-04

Other Changes
  • Reduced per-message allocation on the Async and Ahead concurrency paths by widening the streamly dispatch buffer to 2 * n (the thread bound stays at n). A buffer equal to the thread count throttled worker dispatch and caused dispatch churn that allocated up to +90% on the Async concurrency-levels benchmarks (EP-30). No API or behavior change.
  • Documented the accepted ~126 bytes/message cost of the separate handler and finalizer exception frames that guarantee "always finalize" (EP-31).

0.8.0.0 — 2026-07-04

Breaking Changes

  • New application code should import the Shibuya umbrella module. It re-exports the stable app, handler, batch, retry, policy, metrics, and tracing surface. Shibuya.Core remains as a deprecated compatibility re-export for this release.

  • Runner internals moved under Shibuya.Internal.Runner.* and carry no PVP stability guarantee. The public moves are: Shibuya.Runner.Metrics -> Shibuya.Core.Metrics; Shibuya.Runner.{Master,Supervised,Batcher,BatchProcessor} -> Shibuya.Internal.Runner.*; Shibuya.Runner.{Halt,Ingester} -> Shibuya.Internal.Runner.* as hidden implementation modules. Shibuya.Runner.Serial and Shibuya.Runner.Processor were deleted. Shibuya.Prelude is no longer exposed.

  • AppHandle and Master are now opaque in public modules. Use getAppMaster, getAppMetrics, waitApp, stopApp, and stopAppGracefully instead of record fields. Metrics servers should import Master, getAllMetricsIO, and getProcessorMetricsIO from Shibuya.App.

  • runApp now takes an AppConfig record and validates inboxSize. Invalid sizes return Left (AppConfigInvalid (InvalidInboxSize n)) instead of crashing or stalling.

    -- before
runApp IgnoreFailures 100 processors

-- after
runApp defaultAppConfig processors

-- custom strategy/size
runApp defaultAppConfig {strategy = IgnoreFailures, inboxSize = 100} processors

  • Handlers now receive Message es msg instead of Ingested es msg. Handlers that only use .envelope or .lease need a signature update; handlers can no longer call the adapter finalizer directly. The framework owns finalization and calls AckHandle.finalize after the handler returns or throws. Adapter finalizers should remain idempotent or phase-tracked.

  • BatchHandler likewise receives NonEmpty (Message es msg) instead of NonEmpty (Ingested es msg).

  • Removed dead or misleading surface: HandlerError.HandlerTimeout, RuntimeError.InboxOverflow, StreamStats.dropped, incDropped, the always-zero shibuya_messages_dropped_total Prometheus metric, and Shibuya.Telemetry.Config.

  • Renamed Ordering to OrderingPolicy, eliminating the need for import Prelude hiding (Ordering). runSupervised also takes the ordering policy before Concurrency.

  • BatchingProcessor now rejects PartitionedInOrder combined with Ahead or Async, because batches are scheduled by BatchKey, not Envelope.partition.

  • Renamed Shibuya.Stream.batchStream to chunksOf.

New Features
  • Added mkEnvelope :: MessageId -> msg -> Envelope msg. It fills optional metadata with defaults; set optional fields with record update syntax. This is the recommended construction path because future optional Envelope fields can then be added without forcing adapter code to change, avoiding the kind of major bumps required by 0.5.0.0 and 0.7.0.0.
  • Added mkIngested :: Envelope msg -> AckHandle es -> Ingested es msg.
  • Added Message es msg, the read-only handler view containing envelope and optional lease.
  • PartitionedInOrder with Ahead or Async is now enforced for single-message processors. Messages sharing an Envelope.partition key are processed and acknowledged in arrival order, distinct partitions run concurrently up to the configured bound, and messages with no partition key are unconstrained.
Bug Fixes
  • Handler exceptions on the single-message runner path now finalize the message with AckRetry (RetryDelay 0) using the same bounded finalizer retry helper as batch processing, so adapters observe a disposition instead of silently losing or stranding the delivery. Exhausted finalizer retry now halts the processor loudly with the failed message id.
  • Batch accumulation failures, including exceptions from user-provided batchKey functions, now fail the processor loudly instead of letting the batcher consumer die while the processor reports clean completion.
  • Batch processing now isolates AckHalt: batches already emitted after a halt no longer re-enter the user batch handler, and their messages are finalized with AckRetry (RetryDelay 0) instead of being left unacknowledged.
  • The keyed batch scheduler now keeps its pending queue bounded and owns its reader and worker threads with structured cleanup, preventing unbounded pulls under slow handlers and preventing finalization after forced shutdown returns.
Other Changes
  • Corrected Ahead documentation. It preserves stream-yield order only; handler execution and acknowledgement may complete in any order.
  • TraceHeaders remains a type alias with the same representation as Headers; its docs now clarify that it is only the parsed W3C trace-context projection, while Headers is the complete broker header set.
  • shibuya-pgmq-adapter and shibuya-kafka-adapter should migrate envelope construction to mkEnvelope in their next releases.
  • Removed unused dependencies from shibuya-core: effectful-core, generic-lens, lens, uuid, and vector. Internal unlift imports now use the stable top-level Effectful exports.

0.7.1.0 — 2026-06-15

Bug Fixes
  • Supervised processors now observe their ingester async after draining already-ingested messages. If the source dies with an exception, the processor is marked Failed, its done flag is set, and the exception is rethrown to the supervisor instead of being reported as a clean stream completion.

0.7.0.0 — 2026-06-05

Breaking Changes
  • Envelope gained a headers :: !(Maybe Headers) field carrying every message header the source broker delivered, in order and including duplicates. Direct constructions of Envelope must add the field. Nothing means the adapter does not surface headers; Just [] means it does and the message had none. The new Headers type alias ([(ByteString, ByteString)]) is exported from Shibuya.Core and Shibuya.Core.Types. The W3C trace headers continue to appear in traceContext as before; they now also appear verbatim in headers.

0.6.0.0 — 2026-05-31

Breaking Changes
  • OpenTelemetry messaging spans now emit messaging.operation.type = "process" instead of the deprecated messaging.operation = "process" wire key. The Haskell constant attrMessagingOperation keeps the same name and now resolves to the current semantic-conventions key, so source imports continue to compile. Dashboards, alerts, and trace queries that filter on messaging.operation must be updated to messaging.operation.type.
Other Changes
  • Upgrade OpenTelemetry dependencies to the 1.0 ecosystem: hs-opentelemetry-api ^>= 1.0, hs-opentelemetry-propagator-w3c ^>= 1.0, and test-only hs-opentelemetry-exporter-in-memory ^>= 1.0.
  • Move hs-opentelemetry-semantic-conventions to the latest Haskell generated package available with the 1.0 release, ^>= 1.40, and source Shibuya's generic messaging keys from its typed exports.

0.5.0.0 — 2026-05-05

Breaking Changes
  • Envelope gained an attributes :: !(HashMap Text Attribute) field carrying adapter-supplied OpenTelemetry attributes for the per-message processing span. Direct constructions of Envelope must add the field; pass Data.HashMap.Strict.empty when the adapter has nothing to contribute (the common case). Envelope's NFData instance is now hand-written rather than derived (because Attribute from hs-opentelemetry-api does not ship NFData); the strictness shape is unchanged for every other field.
New Features
  • Shibuya.Runner.Supervised's processOne now applies envelope.attributes to its Consumer-kind span after setting the framework-default messaging.* attributes, so adapter-supplied keys override framework defaults of the same name. This lets broker-aware adapters (Kafka in particular) emit typed attributes (messaging.kafka.destination.partition, messaging.kafka.message.offset) and override the messaging.system default — without opening a second span. The per-record Shibuya.Adapter.Kafka.Tracing.traced wrapper that previously bolted these on can now be removed; see the audit document docs/plans/9-otel-audit-findings.md (Finding F1, P0) for the full motivation.
  • Shibuya.Telemetry.Propagation.currentTraceHeaders :: (Tracing :> es, IOE :> es) => Eff es (Maybe TraceHeaders) looks up the currently-active OTel span (via thread-local context) and encodes its trace context as W3C headers, ready for an adapter to attach to an outgoing message. Returns Nothing when tracing is disabled or there is no active span. The intended call sites are adapter-side DLQ writes (so the failing-consumer's trace links to the resulting DLQ message) and ad-hoc producer paths. See the audit document docs/plans/9-otel-audit-findings.md (Findings F3 and F5).

0.4.0.0 — 2026-04-29

Breaking Changes
  • Envelope gained an attempt :: !(Maybe Attempt) field carrying the adapter's delivery counter (zero-indexed; Nothing if unknown). Direct constructions of Envelope must add the field. The new Attempt newtype is exported from Shibuya.Core and Shibuya.Core.Types.
New Features
  • New module Shibuya.Core.Retry providing BackoffPolicy, Jitter (NoJitter, FullJitter, EqualJitter), defaultBackoffPolicy, the pure evaluator exponentialBackoffPure, the effectful exponentialBackoff, and the handler convenience retryWithBackoff. Handlers can now compute exponentially-growing, jittered retry delays with a single call: retryWithBackoff defaultBackoffPolicy ingested.envelope. Pulls in random ^>=1.2 as a new build-depends (already a transitive dep — ships with GHC). See the module haddock and the new RetrySpec test for usage patterns.
  • A runnable end-to-end demonstration of the new API lives in the sibling shibuya-pgmq-adapter repo at shibuya-pgmq-example/, exposed via the backoff-demo subcommand of shibuya-pgmq-consumer. The plan docs/plans/8-demonstrate-backoff-end-to-end.md records setup instructions and captured transcripts.

0.3.0.0 — 2026-04-24

Version bumped to track the shared release version. No user-visible changes to shibuya-core itself.

0.2.0.0 — 2026-04-22

Breaking Changes
  • Shibuya.Telemetry.Semantic: rename processMessageSpanName :: Text to processSpanName :: Text -> Text. The span name is now built from the destination (processor id), yielding e.g. "shibuya-consumer process", in line with the OpenTelemetry messaging-spans recommendation.
  • Shibuya.Telemetry.Semantic: remove attrMessagingDestinationPartitionId (replaced by the Shibuya-specific attrShibuyaPartition).
  • Shibuya.Telemetry.Semantic: remove eventHandlerException.
New Features
  • Shibuya.Telemetry.Semantic: add attrMessagingOperation and attrShibuyaPartition attribute keys. Messaging attribute keys are now sourced from the typed AttributeKey values exported by OpenTelemetry.SemanticConventions, so upstream renames surface as compile errors rather than silent wire-format drift.
  • Shibuya.Core.Types: add NFData instances for MessageId, Cursor, and Envelope a (when a itself has an NFData instance). Benchmark authors no longer need to declare these as orphans.

0.1.0.0 — 2026-02-24

Initial release.

New Features
  • Multi-queue processing with runApp and QueueProcessor API
  • Backpressure via bounded inbox
  • AckHalt support to stop processing on halt decision
  • Serial, Ahead, and Async concurrent processing modes
  • Policy validation enforcement (StrictInOrder requires Serial)
  • Graceful shutdown with configurable drain timeout
  • NQE-based supervision via Master/Supervisor
  • OpenTelemetry tracing integration with W3C trace context propagation
  • Mock adapter for testing
Bug Fixes
  • Fix race conditions and incomplete cleanup in runner
  • Fix data races in concurrent test handlers using atomicModifyIORef'
Other Changes
  • Consolidate error handling with unified error types
  • Define own SupervisionStrategy type to decouple from NQE
  • Use registerDelay for cleaner timeout handling
  • Replace polling with STM blocking in waitApp