Repository navigation
Daily forecasts are pulled back toward the recent price level and move ~10x as far as the normalisation mean is shifted (evidence for #228 / #307) #422
Description
摘要（中文）： 在日线数据上，Kronos 的预测会被拉回近期价格水平（输入窗口均值附近），20 根 K 线内即抹平最后收盘价与该均值之间的大部分差距。在带有已知向上漂移、本身不含均值回归的随机游走上，Kronos-small 对 20 根 K 线后的平均预测涨跌幅为 -10.4%（30 条序列中 29 条预测下跌），而理论期望涨幅为 +2.1%；真实日线也没有这种回归（见下文第 1 点），因此不能用统计模型常见的反转效应来解释。保持输入 K 线不变、只改变计算归一化统计量所用的数据时，预测的变动幅度约为归一化均值变动的 10 倍（单纯向窗口均值回归最多只有 1 倍），且方向相同，提示模型把归一化统计量当作了关于未来走势的信息。这与 #84 / #227 针对微调代码报告的归一化泄露（统计量包含待预测的 K 线）相符。想跟进 #228 / #307 中的问题：预训练时每个窗口的均值和标准差是用哪些 K 线计算的？损失是在哪些位置上计算的？能否只用每个待预测位置之前的数据计算统计量，重新训练已发布的权重？
Summary
On daily bars, Kronos forecasts are pulled back toward the recent price level: within 20 bars they close most of the gap between the last close and the mean of the input window (with lookbacks of 400 or more, the level is closer to the mean of roughly the last 200-300 bars). This matches the reports of reversing or mean-reverting forecasts in #33, #90, #215 (15-minute bars) and #282, and a Hugging Face discussion on Kronos-small where daily input rising from about 750 to 2700 over the lookback got a flat forecast at about 1500-1750; #375 also measures excess mean reversion in sampled paths. In #2, #33, #46 and #68 it was explained as a reversal effect, which #2, #33 and #68 describe as common to statistical models. Three checks point to an artefact of the model instead:
- Real prices do not revert like this. Over 3,012 non-overlapping 20-bar targets before 2024-07 on the 9 stocks and ETFs used below (lookback 400), the realised pull is +0.02 window-std per unit of distance from the window mean, and windows ending 1.5 std or more above their mean went on to rise 61% of the time. On the 2024-26 hindcasts below, Kronos-small pulls -0.82 where the realised pull was -0.18, and its forecasts lose to a flat forecast (median MAPE over the 20 bars 7.8% vs 3.1%).
- Random walks with a known upward drift (no reversion by construction): Kronos-small forecasts -10.4% over 20 bars on average (down in 29 of 30) where the expected change is +2.1%.
- Holding the input bars fixed and changing only the rows the normalisation statistics are computed over moves the forecast about 10x as far as the normalisation mean moved, in the same direction. A model that merely reverts toward the window mean would move at most 1x. The std moves too, but on its own it has no measurable effect (mirrored test below). The forecast behaves as if it reads the normalisation statistics as information about the future.
This is what we would expect from training on windows normalised with statistics that include the bars being predicted (normalised that way, a window averages zero, so bars still to come must offset those already seen). The fine-tuning code did this (#84, and #227, which names finetune_csv/finetune_base_model.py): #234 closed #227 but changed only finetune/dataset.py to lookback-only statistics, and finetune_csv/finetune_base_model.py#L125 still does it (open PR #263). The released checkpoints (uploaded 2025-06/07) predate those changes, and the pretraining code is not public, so we cannot check this directly. #1 says the statistics are computed over the lookback window of each input sequence; with a next-token loss on every position (finetune/train_predictor.py#L104-L109, finetune_csv/finetune_base_model.py#L288-L292) such statistics still include bars that earlier positions are trained to predict; #84's proposal to start the loss at the lookback boundary avoids this. #228 and #307 ask whether the released checkpoints are affected; this is evidence that they are. Two questions: (1) In pretraining, which rows were the mean/std computed over, and which positions did the loss score? (2) Could the checkpoints be retrained with statistics only from bars before each scored position?
Reproduction (synthetic data, CPU)
Save as repro_reversion.py in the repo root and run python repro_reversion.py (about 4-5 minutes with 8 threads; numbers may differ slightly on other machines). Settings as in the README example (lookback 400, T 1.0, top_p 0.9), 5 samples averaged, models in eval mode (see #415).
repro_reversion.py
# Run from the Kronos repo root (CPU; about 4-5 minutes with 8 threads). Synthetic data only; models in eval mode.
import os
import numpy as np
import pandas as pd
import torch
from model import Kronos, KronosPredictor, KronosTokenizer
from model.kronos import calc_time_stamps
torch.set_num_threads(min(8, os.cpu_count() or 1))
tokenizer = KronosTokenizer.from_pretrained("NeoQuasar/Kronos-Tokenizer-base").eval()
model = Kronos.from_pretrained("NeoQuasar/Kronos-small").eval()
predictor = KronosPredictor(model, tokenizer, device="cpu", max_context=512)
DRIFT = 0.001 # +0.1% a day in log price, 1% daily noise: expected 20-bar change exp(20*DRIFT + 20*0.01**2/2) - 1 = +2.1%
def walk(seed, n=420):
"""Geometric random walk with a known upward drift and 1% daily noise (no mean reversion by construction)."""
rng = np.random.default_rng(seed)
close = 100 * np.exp(np.cumsum(DRIFT + 0.01 * rng.standard_normal(n)))
prev = np.r_[close[0], close[:-1]]
spread = np.abs(0.005 * rng.standard_normal(n)) * close
return pd.DataFrame({"timestamps": pd.bdate_range("2021-01-04", periods=n), "open": prev,
"high": np.maximum(prev, close) + spread, "low": np.minimum(prev, close) - spread,
"close": close, "volume": rng.integers(5_000_000, 15_000_000, n).astype(float)})
def forecast_close(x, x_ts, y_ts, pad):
"""KronosPredictor.predict step for step, except that mean/std are computed over the input plus `pad` copies
of its last bar. pad=0 is exactly predict(). Returns the bar-20 forecast close and the close mean used."""
df = x.copy()
df["amount"] = df["volume"] * df[["open", "high", "low", "close"]].mean(axis=1)
arr = df[["open", "high", "low", "close", "volume", "amount"]].values.astype(np.float32)
basis = np.concatenate([arr, np.repeat(arr[-1:], pad, axis=0)]) if pad else arr
mean, std = np.mean(basis, axis=0), np.std(basis, axis=0)
xn = np.clip((arr - mean) / (std + 1e-5), -5, 5)
xs = calc_time_stamps(x_ts).values.astype(np.float32)
ys = calc_time_stamps(y_ts).values.astype(np.float32)
out = predictor.generate(xn[None], xs[None], ys[None], 20, 1.0, 0, 0.9, 5, False).squeeze(0)
return float(out[-1, 3] * (std[3] + 1e-5) + mean[3]), float(mean[3])
rows = []
for seed in range(1, 31):
df = walk(seed)
x, y = df.iloc[:400], df.iloc[400:]
x_df = x[["open", "high", "low", "close", "volume"]].reset_index(drop=True)
x_ts, y_ts = x["timestamps"].reset_index(drop=True), y["timestamps"].reset_index(drop=True)
torch.manual_seed(seed)
std_close, std_mean = forecast_close(x_df, x_ts, y_ts, 0)
torch.manual_seed(seed)
pad_close, pad_mean = forecast_close(x_df, x_ts, y_ts, 20)
c = x["close"].to_numpy()
last, sd = c[-1], c.std()
rows.append(((last - c.mean()) / sd, (std_close - last) / sd, std_close / last - 1,
y["close"].iloc[-1] / last - 1, pad_mean - std_mean, pad_close - std_close))
z, dz_pred, pred_chg, true_chg, mean_shift, close_shift = map(np.array, zip(*rows))
print(f"up-drifting random walks (n={len(z)}, lookback 400, 20 bars, T 1.0, top_p 0.9, mean of 5 samples):")
print(f" mean forecast 20-bar change {pred_chg.mean():+.1%}, down in {(pred_chg < 0).sum()} of {len(z)}; true expected "
f"{np.exp(20 * DRIFT + 20 * 0.01 ** 2 / 2) - 1:+.1%} (realised mean {true_chg.mean():+.1%}); median z_last {np.median(z):+.2f}")
print(f" slope of the forecast move on z_last: {np.polyfit(z, dz_pred, 1)[0]:+.2f} window-std per unit z (a random walk has 0)")
print(f" adding 20 copies of the last bar to the rows the mean/std are computed over (same input and seed) moves the bar-20 "
f"forecast {np.polyfit(mean_shift, close_shift, 1)[0]:.1f}x the shift of the normalisation mean (reverting toward the "
f"mean alone would give at most 1x)")Output (Python 3.12, torch 2.14.1 CPU, huggingface_hub 0.33.1, numpy 2.5.4, pandas 2.2.2):
up-drifting random walks (n=30, lookback 400, 20 bars, T 1.0, top_p 0.9, mean of 5 samples):
mean forecast 20-bar change -10.4%, down in 29 of 30; true expected +2.1% (realised mean +3.3%); median z_last +1.83
slope of the forecast move on z_last: -1.02 window-std per unit z (a random walk has 0)
adding 20 copies of the last bar to the rows the mean/std are computed over (same input and seed) moves the bar-20 forecast 10.9x the shift of the normalisation mean (reverting toward the mean alone would give at most 1x)
Same on real daily data
Kronos-small, same settings, 20-bar hindcasts on Yahoo Finance adjusted daily bars. "Slope" = forecast move over 20 bars in window-std units per unit of z_last = (last close - window mean) / window std; 0 means no pull, -1 means all the way back to the mean. Intervals resample target dates.
- SPY, QQQ, AAPL, MSFT, NVDA, 29 non-overlapping 20-bar targets each from 2024-07-01 (the 400-bar inputs overlap): 123 of 145 forecasts down, mean -10.2% vs +1.7% realised; slope -0.82 (95% CI -0.93 to -0.73); the realised slope on the same windows is -0.18 (CI -0.37 to +0.04).
- Both directions: in the 2000-02 bear market, the 75 windows ending more than 1 std below their mean were forecast to rise by a median +10% (mean +25%; 58 of 75 up) where +0.2% followed on average; slope over that period -0.56.
- Across settings: Kronos-mini -0.46, Kronos-base -1.15; lookbacks 64 to 512 (-0.98 at 64, -0.80 to -0.88 from 128 to 512); targets in 2012-14 (-0.67) and 2019-21 (-0.82); TLT, GLD, XLE, EFA (-0.76); T 0.1, nearly greedy (-0.80); top_p 1.0 (-0.86).
- Normalisation tests on the same 145 windows and seeds (input bars unchanged; the normalised input moves only about 0.06 std): padding the statistics with 20 copies of the last bar, with 20 mirrored values (2 x mean - last) or, as a probe, with the true next 20 bars moves the forecast 9-11x the shift of the normalisation mean, in its direction. The mirrored pad has the same std as the last-bar pad but the opposite mean shift, and it steepens the pull (slope -1.32 vs -0.82): the forecast follows the mean shift, not the std change. Averaging the last-bar and mirrored runs cancels their opposite mean shifts but keeps their identical std, and that average matches the unpadded forecast (slope difference -0.02, 95% CI -0.06 to +0.01), so the std change on its own has no measurable effect.
- Eval mode (fix: call .eval() on tokenizer and model in KronosPredictor #382) changes these forecasts only slightly (per-window correlation 0.96) and does not remove the pull.
Suggested direction
- Retrain or fine-tune with statistics computed only from bars before each scored position (or score only positions after the statistics window, as Fix: Remove window-normalization leakage in AR training (causal lookback normalization) #84 proposes), then re-run the random-walk check above: forecasts should stop falling on average and should move at most 1x with the normalisation mean.
- Until then, document that daily forecasts are pulled toward the recent price level, so in a trend a longer lookback (up to about 400 bars) gives a larger reversal (mean 20-bar forecast on the 145 windows: about -1.6%, -3.5%, -7.5%, -10.2% and -9.7% at lookbacks 64, 128, 256, 400 and 512); this is the effect asked about in Lookback and prediction #90.
Activity
hatomwas commentedon Oct 29, 2026
Follow-up: we tested the "Suggested direction" above with a small pre-registered finetune of the released Kronos-mini, in two rounds. Round 1 was inconclusive, as pre-registered. Round 2 (a second seed plus a control arm) tested whether round 1's large difference between the arms, which round 1's rule did not use, holds up; its pre-registered outcome is "replicated" (details below). Each round's decision rules were committed to our (private) repository before its code.
The arms differ only in the rows the training mean/std are computed over:
predict()does (model/kronos.py#L544).Every model was then evaluated in eval mode through the unmodified
predict(), except two probes built on its code: the normalisation-mean test adds 20 copies of the last bar to the statistics (as in the issue's repro), and the #423 check below patches the decode_s2 positions.Pre-registered outcomes:
Slope as in the issue (0 = no pull). Cells with "/" give round 1 / round 2:
Round 2, slopes paired on the same windows and sampling seeds: C minus B is +0.44 to +1.48 on all five sets (every 95% interval above +0.3; the five sets are the walks plus two asset groups, each at two lookbacks that share target dates); A minus C is -0.004 to +0.014 on the four real-data sets (every interval includes 0) and +0.08 [+0.02, +0.16] on the driftless walks. Both checks under "Suggested direction" (the up-drift walks stop falling; the response to the normalisation mean is at most about 1x) came out as predicted for A and C, and not for B. A and C under-react to the drift, though: in round 2 it moved their forecasts by +1.3 and +1.0 points (expected +2.0).
Setup: 5,000 steps from the released Kronos-mini, batch 16, AdamW and OneCycleLR as in
finetune/; daily bars of 35 ETFs, all training targets before 2023-07-01; one lookback of 64/128/256/400 per batch; the loss scores only the 20 bars after the lookback (as #84 proposes), with s2 teacher-forced on the true s1. Evaluation: T 1.0, top_p 0.9, 5 samples. Our training code is not public; this is the only place the arms differ:What this does not show:
predict()with the bug in place. With the fix suggested in decode_s2: rotary positions collapse to 0 in the single-query cross-attention (train/inference mismatch) #423, the US lookback-400 slope moves from +0.26 to +0.08 for A, from +0.26 to +0.04 for C and from -0.18 to -0.33 for B, but only from -0.46 to -0.45 for the released model: the finetuned models are far more sensitive to the bug. C minus B stays at +0.36 [+0.26, +0.48]. In a post hoc run on round 1's A, the fix left the lookback-64 tilt in place (US +0.29, ETFs +0.33); round 2 ran the fix at lookback 400 only.finetune/recipe (lookback-only statistics since Fix data leakage in normalization window (#227) #234, with the loss on every position) was not tested. A and B were each trained twice (the seed changes the sampled windows, lookbacks and dropout, not the starting weights) and C once; the intervals resample test windows, not training runs. The closest pre-registered call was B's gain: its lower bound was 6.0 against a threshold of 5, while B's gain moved by 1.3 between its two runs (9.1x, 7.8x); below 5, round 2's outcome would have been "mixed". The data (Yahoo Finance) and the finetuned weights are not shared.Question 1 of the issue is still open, and a finetune cannot answer it: when the released checkpoints were pretrained, were each window's mean/std computed only over rows before the first position the loss scored, or did they include scored rows (for example the whole window, with a next-token loss on every position)?
中文： 我们从已发布的 Kronos-mini 出发，做了两轮预注册的小规模微调。第一轮结果为不确定（inconclusive）；第二轮（换一个随机种子，并加入对照组）检验第一轮中各组之间的明显差异能否复现，预注册的结论为“replicated”（复现）。三组中，只有训练时归一化统计量包含待预测的 20 根 K 线的那一组保留了向窗口均值的回拉；只用回看窗口、或用回看窗口加上其之前的 20 根 K 线计算统计量的两组，回拉都消失了。我们没有测试不含未来 K 线、但像前者一样偏向最后收盘价的统计量。三组都没有通过我们的无回拉检验：回拉消失的两组在 2024-26 年的真实数据上反而倾向于远离窗口均值，预测误差也不比直接沿用最后收盘价的朴素预测小。这是微调结果，不能说明已发布权重的预训练方式。问题：预训练已发布的权重时，每个窗口的均值和标准差是只用第一个计入损失的位置之前的 K 线计算的，还是也包含了计入损失的 K 线（例如对整个窗口计算统计量，并在每个位置计算 next-token 损失）？