The integration problem in plain terms
The question we hear most often from ML platform engineers is some version of: "If I drop your card into a server, how much of my inference stack do I need to rewrite?" It is a fair question and one we spent considerable time designing around. The short answer is: almost nothing in the model code, and a driver + SDK layer that handles the translation.
The longer answer requires understanding where the photonic co-processor actually sits in the inference stack, and what it does and does not replace.
Where the co-processor fits in the inference stack
Current transformer inference workflows on NVIDIA hardware generally run through PyTorch or JAX calling cuBLAS or cuDNN kernels for the matrix multiplications. The GPU handles everything: attention, projection layers, feedforward blocks, normalization. The entire model runs on the GPU.
The GSK-1 is not trying to replace the GPU. It targets the linear projection layers: the weight-multiplied projections in attention (Q, K, V, and output projections) and the feedforward block linear layers. These layers share a common structure: they are dense matrix-vector products where the weight matrix is large, fixed for a given model, and fetched from memory on every forward pass.
The co-processor sits on a PCIe Gen 4 x16 slot alongside the GPU. When inference is running, the host CPU dispatches projection-layer computations to the photonic chip via the driver layer while the GPU handles the non-linear operations (softmax, layer norm, activation functions). For a typical large transformer, the projection layers account for the majority of MAC operations per token, so the co-processor handles a substantial fraction of the compute, and the GPU handles the remainder.
The matmul interception layer
We designed the SDK to intercept at the level of the linear layer, not the individual kernel call. Here is the basic pattern in Python:
import greatsky
# At model initialization
proc = greatsky.PhotonicProcessor()
proc.attach()
# Program weights once per model load
for name, module in model.named_modules():
if isinstance(module, torch.nn.Linear):
if proc.eligible(module.weight.shape):
proc.program_layer(name, module.weight)
proc.intercept(module)
# Inference loop runs unchanged
output = model(input_ids)After the proc.intercept(module) call, the SDK replaces the module's forward method with one that routes the matrix multiply to the photonic hardware. From PyTorch's perspective, the module still returns a tensor of the correct shape and dtype. The model code, the training loop if it is running in eval mode, and any downstream consumers of the output see no change.
The tensor is transferred off the GPU via PCIe to host pinned memory, sent to the co-processor's input buffer, the photonic matrix multiply runs, the result comes back to pinned memory, and is copied back to the GPU for the next operation. The round-trip PCIe latency is the main latency cost we add. We spend a lot of time in driver optimization making sure that host-device copies are pipelined and that we are not stalling the GPU unnecessarily while the photonic compute runs.
Driver architecture and the PCIe bandwidth budget
PCIe Gen 4 x16 gives roughly 32 GB/s of bidirectional bandwidth in practice after encoding overhead. For a 4096-dimensional projection with float16 activations, one forward-pass input vector is 8 KB. A batch of 128 tokens is about 1 MB of input data, and the same amount of output data. That is a 2 MB round-trip over PCIe for each projection layer call.
A large transformer might call 96 such layers per forward pass. At batch 128, that is roughly 192 MB of PCIe traffic per forward pass. At 32 GB/s PCIe bandwidth, the transfer budget is around 6 milliseconds. For throughput-optimized inference running in large batches, that fits comfortably within the time budget. For single-token latency-critical use cases, it does not, and we are direct about this in our integration documentation.
The driver maintains a command queue and handles interrupt-driven completion notification so the host CPU is not polling. We use Linux kernel DMA coherent allocation for the input/output buffers to avoid cache coherency penalties. The driver is tested on Linux 5.15 and later kernels; there is no Windows driver yet, which is not typically a constraint for data-center inference workloads.
Handling quantized models
Most production inference stacks run quantized models: INT8, INT4, or mixed precision. The photonic processor operates with analog optical signals, not digital bit representations, so the quantization interaction requires some care.
We handle INT8 activations by dequantizing to FP16 before transferring to the co-processor input buffer. The photonic computation runs in what we call 6-to-8 effective bits of analog precision. The result is requantized to INT8 before returning to the GPU. This adds a small quantization error on top of the model's existing quantization error. In our bench testing on a range of transformer architectures (BERT-style encoders and GPT-style decoder stacks with varying width), the perplexity increase from photonic quantization on top of INT8 weight quantization has been within acceptable bounds for inference workloads, though we are still characterizing this systematically and would not describe it as a closed research question yet.
INT4 models require more care. At 4-bit weight quantization, the signal-to-noise interaction between digital quantization and optical analog resolution can compound in ways that require architecture-specific calibration. We recommend running INT4 inference through our calibration pipeline before production deployment with the co-processor.
What the GPU still does
We are not replacing the GPU. The GPU still runs all non-linear operations: softmax, ReLU, GeLU, SiLU, layer normalization, RMSNorm, any attention masking or causal masking operations, and the KV-cache management. The GPU also handles embedding lookups, output logit computation, and sampling. For a standard causal language model doing autoregressive generation, the GPU is continuously active managing the KV cache and doing the per-token non-linear work while the co-processor handles the weight-stationary projection compute.
The model for thinking about this is not "GPU vs photonic processor" but rather "GPU with a dedicated matrix-multiply coprocessor handling the large projection layers." The GPU is still essential. It just no longer needs to fetch 140 GB of weights from HBM on every forward pass for the layers we handle, because those weights are encoded optically in the co-processor.
If your inference workload is already compute-bound on the GPU (small models, short sequences, very low batch sizes), adding the co-processor does not help and may hurt latency. The integration is most beneficial when your inference nodes are memory-bandwidth-bound on large models with large batch sizes, which describes most large-scale inference deployments running today.