Initial commit: BTC 1h prediction model (53.10% accuracy)

🎯 项目成果:
- 准确率: 53.10% (2026年测试集)
- 任务类型: 二分类 (预测涨/跌)
- 特征: OHLCV + RSI + MACD (9个特征)
- 训练数据: 2020-2025年BTC小时数据 (52,351序列)
- 测试数据: 2026年1-29日 (484序列)

📊 性能指标:
- 精确率: 51.79%
- 召回率: 49.36%
- F1分数: 0.5054
- 预测分布: 非常平衡 (偏差仅-11个)

🔬 实验历史:
- 尝试了5种方案 (回归/分类, 不同数据划分)
- 分类任务优于回归任务 (53.10% vs 49.5%)
- 全量训练避免了过拟合问题

📦 主要模型文件:
- models/checkpoints/dlinear_classification_2026/
  - best_model.pt (最佳模型)
  - preprocessor.pkl (预处理器)
  - test_2026_results.json (测试结果)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
crypto_predictorandClaude Sonnet 4.5 committed 2026-01-29 21:27:15 +08:00
commit 9bbdc8d42b
82 files changed
+8742

No files matched your search

+10
View File
@@ -0,0 +1,10 @@
{
"permissions": {
"allow": [
"Bash(/Users/xxz/miniforge3/bin/conda run:*)",
"Bash(tee:*)",
"Bash(python:*)",
"Bash(source:*)"
]
}
}
+32
View File
@@ -0,0 +1,32 @@
# Binance API (public data - no keys needed)
BINANCE_API_URL=https://api.binance.com
# 代理配置 (SOCKS5)
SOCKS5_PROXY=socks5://127.0.0.1:7890
USE_PROXY=true
# Data Configuration
DATA_CACHE_DIR=./data/cache
LOOKBACK_1H=168 # 1 week for 1h data
LOOKBACK_15M=672 # 1 week for 15m data
FORECAST_HORIZON=24 # Predict 24 periods ahead
# Training Configuration
BATCH_SIZE=64
LEARNING_RATE=0.001
MAX_EPOCHS=100
EARLY_STOPPING_PATIENCE=10
TRAIN_SPLIT=0.7
VAL_SPLIT=0.15
TEST_SPLIT=0.15
# Model Paths
MODEL_DIR=./models/checkpoints
ONNX_DIR=./models/onnx
# API Configuration
API_PORT=5000
API_HOST=0.0.0.0
# Classification Threshold
PRICE_CHANGE_THRESHOLD=0.015 # 1.5% threshold for binary labels
+32
View File
@@ -0,0 +1,32 @@
# Binance API (public data - no keys needed)
BINANCE_API_URL=https://api.binance.com
# 代理配置 (SOCKS5)
SOCKS5_PROXY=socks5://127.0.0.1:7890
USE_PROXY=true
# Data Configuration
DATA_CACHE_DIR=./data/cache
LOOKBACK_1H=168 # 1 week for 1h data
LOOKBACK_15M=672 # 1 week for 15m data
FORECAST_HORIZON=24 # Predict 24 periods ahead
# Training Configuration
BATCH_SIZE=64
LEARNING_RATE=0.001
MAX_EPOCHS=100
EARLY_STOPPING_PATIENCE=10
TRAIN_SPLIT=0.7
VAL_SPLIT=0.15
TEST_SPLIT=0.15
# Model Paths
MODEL_DIR=./models/checkpoints
ONNX_DIR=./models/onnx
# API Configuration
API_PORT=5000
API_HOST=0.0.0.0
# Classification Threshold
PRICE_CHANGE_THRESHOLD=0.015 # 1.5% threshold for binary labels
+40
View File
@@ -0,0 +1,40 @@
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
venv/
.venv/
env/
ENV/
# 数据和缓存
data/cache/*.csv
*.db
# 模型检查点(保留最佳模型,忽略训练过程文件)
models/checkpoints/*/epoch_*.pt
models/checkpoints/*/training_history.json
# Jupyter
.ipynb_checkpoints
*.ipynb
# IDE
.vscode/
.idea/
*.swp
*.swo
# 系统文件
.DS_Store
Thumbs.db
# 日志
*.log
logs/
# 临时文件
*.tmp
.cache/
+141
View File
@@ -0,0 +1,141 @@
# Crypto Predictor
基于深度学习的加密货币价格预测系统,使用DLinear模型预测BTC未来1小时涨跌方向。
## 🎯 项目概况
**当前最佳模型性能**:
- **准确率**: 53.10% ✅ (在2026年测试集上)
- **任务类型**: 二分类(预测涨/跌)
- **特征**: OHLCV + RSI + MACD (9个特征)
- **训练数据**: 2020-2025年BTC小时数据(52,351个序列)
- **测试数据**: 2026年1-29日(484个序列)
## 📊 模型性能指标
```
准确率 (Accuracy): 53.10%
精确率 (Precision): 51.79%
召回率 (Recall): 49.36%
F1分数: 0.5054
混淆矩阵:
预测涨且真涨: 116 (24.0%)
预测跌且真跌: 141 (29.1%)
预测涨但真跌: 108 (22.3%)
预测跌但真涨: 119 (24.6%)
预测分布:
预测上涨: 224 (46.3%)
预测下跌: 260 (53.7%)
实际上涨: 235 (48.6%)
实际下跌: 249 (51.4%)
偏差: -11个 (非常平衡)
```
## 🏗️ 项目结构
```
crypto_predictor/
├── data/
│ ├── fetcher.py # Binance数据获取
│ ├── preprocessor.py # 数据预处理
│ └── dataset.py # PyTorch数据集
├── models/
│ ├── base.py # 基础模型类
│ ├── dlinear/
│ │ └── model.py # DLinear模型实现
│ └── checkpoints/
│ └── dlinear_classification_2026/ # 最佳模型(53.10%)
│ ├── best_model.pt
│ ├── preprocessor.pkl
│ └── test_2026_results.json
├── training/
│ ├── trainer.py # 训练器
│ ├── config.py # 训练配置
│ └── evaluator.py # 评估器
├── train_classification_full.py # 分类任务训练脚本
├── test_2026_classification.py # 2026年测试脚本
└── requirements.txt
```
## 🚀 快速开始
### 1. 环境准备
```bash
# 创建conda环境
conda create -n crypto_pred python=3.12
conda activate crypto_pred
# 安装依赖
pip install -r requirements.txt
```
### 2. 训练模型
```bash
# 使用2020-2025年数据训练分类模型
python train_classification_full.py
```
### 3. 测试模型
```bash
# 在2026年数据上测试
python test_2026_classification.py
```
## 📈 使用最佳模型进行预测
```python
import torch
import pickle
from data.fetcher import BinanceDataFetcher
from models.dlinear.model import DLinear
# 加载模型
device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu')
checkpoint = torch.load('./models/checkpoints/dlinear_classification_2026/best_model.pt',
map_location=device)
model = DLinear(checkpoint['config'])
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()
# 加载预处理器
with open('./models/checkpoints/dlinear_classification_2026/preprocessor.pkl', 'rb') as f:
preprocessor = pickle.load(f)
# 获取最新数据并预测
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h', use_cache=False)
X, y, _ = preprocessor.preprocess(df, fit=False)
# 预测
with torch.no_grad():
logits = model(torch.FloatTensor(X[-1:]).to(device))
prob = torch.sigmoid(logits).item()
prediction = "上涨" if prob > 0.5 else "下跌"
print(f"预测: {prediction} (概率: {prob:.2%})")
```
## 🔬 实验历史
| 方案 | 任务类型 | 数据划分 | 准确率 | 主要问题 |
|------|---------|---------|--------|---------|
| 1 | 回归(收益率) | 95:5 | 50.23% | 接近随机 |
| 2 | 回归(收益率) | 直接2026 | 49.48% | 偏向预测上涨 |
| 3 | 回归(价格) | 直接2026 | 49.48% | MAPE 0.42% |
| 4 | 回归(价格) | 8:1:1 | 49.64% | 90%预测下跌 |
| **5** | **分类(涨跌)** | **全量训练** | **53.10%** ✅ | **预测平衡** |
## ⚠️ 注意事项
1. **53.10%准确率说明**:仅比随机猜测高3.1%,适合小仓位测试
2. **1小时预测的局限性**:价格变化接近随机游走,技术指标效果有限
3. **数据要求**:需要至少169小时的历史数据
## 📄 许可
MIT License
+195
View File
@@ -0,0 +1,195 @@
"""详细分析训练/验证/测试集的分配方式"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import pandas as pd
import numpy as np
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
def analyze_split_strategy():
print("="*80)
print("数据集分割策略详细分析")
print("="*80)
# 获取数据
print("\n[1/4] 加载2020-2025年完整数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2020-01-01',
end_date='2025-12-31',
use_cache=True
)
print(f"✓ 原始数据: {len(df)} 行")
print(f"✓ 时间范围: {df['timestamp'].min()} 至 {df['timestamp'].max()}")
# 预处理
print("\n[2/4] 数据预处理...")
preprocessor = CryptoPreprocessor(
lookback=168,
forecast_horizon=1,
price_change_threshold=0.0
)
X, y, processed_df = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
print(f"✓ 序列数量: {len(X)}")
print(f"✓ 每个序列: {X.shape[1]}小时 × {X.shape[2]}特征")
# 分析split_data函数的逻辑
print("\n[3/4] 分析数据分割逻辑...")
print(f"\n📋 split_data函数的分割方式:")
print(f" train_split = 0.7 (70%)")
print(f" val_split = 0.15 (15%)")
print(f" test_split = 0.15 (15%)")
print(f"\n 分割方式: 时间序列顺序分割(非随机打乱)")
print(f" 计算方式:")
print(f" n = len(X) # 总序列数")
print(f" train_end = int(n * 0.7)")
print(f" val_end = int(n * 0.85)")
print(f" 训练集: X[0:train_end]")
print(f" 验证集: X[train_end:val_end]")
print(f" 测试集: X[val_end:]")
# 执行分割
train_data, val_data, test_data = split_data(X, y, train_split=0.7, val_split=0.15)
n = len(X)
train_end_idx = int(n * 0.7)
val_end_idx = int(n * 0.85)
print(f"\n[4/4] 详细分割结果分析...")
print(f"\n总序列数: {n:,}")
print(f"分割点:")
print(f" 训练集结束: 第 {train_end_idx:,} 个序列")
print(f" 验证集结束: 第 {val_end_idx:,} 个序列")
# 计算每个数据集对应的原始数据索引
# 注意:由于lookback=168,序列创建时有偏移
lookback = 168
# 训练集对应的数据范围
train_start_row = lookback # 第一个序列从lookback开始
train_end_row = train_end_idx + lookback
# 验证集对应的数据范围
val_start_row = train_end_idx + lookback
val_end_row = val_end_idx + lookback
# 测试集对应的数据范围
test_start_row = val_end_idx + lookback
test_end_row = len(processed_df)
print(f"\n📊 各数据集对应的原始数据行:")
print(f"\n训练集:")
print(f" 序列索引: 0 到 {train_end_idx:,}")
print(f" 数据行范围: {train_start_row:,} 到 {train_end_row:,}")
train_df = processed_df.iloc[train_start_row:train_end_row]
print(f" 时间范围: {train_df['timestamp'].min()} 至 {train_df['timestamp'].max()}")
train_return = (train_df['close'].iloc[-1] - train_df['close'].iloc[0]) / train_df['close'].iloc[0]
print(f" 价格: ${train_df['close'].iloc[0]:,.0f} → ${train_df['close'].iloc[-1]:,.0f}")
print(f" 收益率: {train_return*100:+.2f}%")
print(f"\n验证集:")
print(f" 序列索引: {train_end_idx:,} 到 {val_end_idx:,}")
print(f" 数据行范围: {val_start_row:,} 到 {val_end_row:,}")
val_df = processed_df.iloc[val_start_row:val_end_row]
print(f" 时间范围: {val_df['timestamp'].min()} 至 {val_df['timestamp'].max()}")
val_return = (val_df['close'].iloc[-1] - val_df['close'].iloc[0]) / val_df['close'].iloc[0]
print(f" 价格: ${val_df['close'].iloc[0]:,.0f} → ${val_df['close'].iloc[-1]:,.0f}")
print(f" 收益率: {val_return*100:+.2f}%")
print(f"\n测试集:")
print(f" 序列索引: {val_end_idx:,} 到 {n:,}")
print(f" 数据行范围: {test_start_row:,} 到 {test_end_row:,}")
test_df = processed_df.iloc[test_start_row:test_end_row]
print(f" 时间范围: {test_df['timestamp'].min()} 至 {test_df['timestamp'].max()}")
test_return = (test_df['close'].iloc[-1] - test_df['close'].iloc[0]) / test_df['close'].iloc[0]
print(f" 价格: ${test_df['close'].iloc[0]:,.0f} → ${test_df['close'].iloc[-1]:,.0f}")
print(f" 收益率: {test_return*100:+.2f}%")
# 可视化时间线
print(f"\n" + "="*80)
print("📅 时间线可视化")
print("="*80)
print(f"\n2020━━━━━━━━━━━━训练集━━━━━━━━━━━━━2024━━验证━━2025━━测试━━2025")
print(f" ")
print(f"牛市+825% 牛市+34% 下跌-8.5%")
print(f" ")
# 分析问题
print(f"\n" + "="*80)
print("🔍 关键发现")
print("="*80)
print(f"\n✅ 分割方式:正确")
print(f" - 时间序列顺序分割(未打乱)")
print(f" - 训练在前,验证在中,测试在后")
print(f" - 避免了数据泄露")
print(f"\n⚠️ 问题:分布严重不匹配")
print(f" - 训练集(2020-2024初):+825% 🚀 极端牛市")
print(f" - 验证集(2024): +34% 📈 牛市延续")
print(f" - 测试集(2025): -8.5% 📉 回调下跌")
print(f"\n💡 这导致:")
print(f" 1. 模型在牛市中训练 → 学会'牛市思维'")
print(f" 2. 验证集也是牛市 → 验证损失正常,未能发现问题")
print(f" 3. 测试集却是熊市 → 泛化性能差")
# 标签分布分析
print(f"\n📊 标签分布分析:")
train_up_ratio = (train_data[1] == 1).sum() / len(train_data[1])
val_up_ratio = (val_data[1] == 1).sum() / len(val_data[1])
test_up_ratio = (test_data[1] == 1).sum() / len(test_data[1])
print(f"\n 训练集上涨比例: {train_up_ratio*100:.1f}%")
print(f" 验证集上涨比例: {val_up_ratio*100:.1f}%")
print(f" 测试集上涨比例: {test_up_ratio*100:.1f}%")
if abs(train_up_ratio - test_up_ratio) > 0.05:
print(f"\n ⚠️ 训练集和测试集的上涨比例差异: {abs(train_up_ratio - test_up_ratio)*100:.1f}%")
print(f" 这说明分布不一致!")
# 建议
print(f"\n" + "="*80)
print("💡 解决方案建议")
print("="*80)
print(f"\n方案1: Walk-Forward验证(滚动窗口)⭐⭐⭐⭐⭐")
print(f" - 不是70/15/15一次性分割")
print(f" - 而是多次滚动分割:")
print(f" 第1次: 2020-2022训练 → 2023Q1测试")
print(f" 第2次: 2020-2023训练 → 2023Q2测试")
print(f" 第3次: 2020-2023训练 → 2023Q3测试")
print(f" ...")
print(f" - 平均所有测试结果")
print(f"\n方案2: K-Fold时间序列交叉验证 ⭐⭐⭐⭐")
print(f" - TimeSeriesSplit (保持时间顺序)")
print(f" - 多个fold,每个fold包含不同市场环境")
print(f"\n方案3: 分层采样(按市场环境)⭐⭐⭐")
print(f" - 识别牛市/熊市/震荡期")
print(f" - 确保训练/验证/测试都包含各种环境")
print(f" - 但要保持时间顺序")
print(f"\n方案4: 使用更近期的数据作为测试集 ⭐⭐")
print(f" - 例如:2020-2024训练 → 2025测试")
print(f" - 目前就是这样,但问题是2025恰好是下跌期")
print(f"\n当前问题的本质:")
print(f" ❌ 不是分割方法的问题")
print(f" ❌ 而是时间序列预测的固有难题:")
print(f" '过去的牛市'无法预测'未来的熊市'")
print(f" 除非训练数据包含足够多的熊市样本")
print("\n" + "="*80)
if __name__ == "__main__":
analyze_split_strategy()
+244
View File
@@ -0,0 +1,244 @@
"""详细对比二分类和回归任务在加密货币预测中的差异"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
import pandas as pd
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor
def compare_classification_vs_regression():
print("="*80)
print("二分类 vs 回归任务详细对比")
print("="*80)
# 获取少量数据用于演示
print("\n[1/3] 获取示例数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h', use_cache=True)
df_sample = df.tail(200).copy() # 只取最近200小时
print(f"✓ 样本数据: {len(df_sample)} 行")
print(f"✓ 时间范围: {df_sample['timestamp'].min()} 至 {df_sample['timestamp'].max()}")
# 对比两种任务
print("\n[2/3] 对比两种任务的标签生成...")
# 二分类任务
print("\n" + "="*80)
print("📊 任务1: 二分类(Binary Classification)")
print("="*80)
preprocessor_cls = CryptoPreprocessor(
lookback=24,
forecast_horizon=1,
price_change_threshold=0.0
)
# 创建标签
df_sample['future_price'] = df_sample['close'].shift(-1)
df_sample['future_return'] = (df_sample['future_price'] - df_sample['close']) / df_sample['close']
df_sample['label_classification'] = (df_sample['future_return'] > 0).astype(int)
print(f"\n目标: 预测价格会上涨(1)还是下跌(0)")
print(f"\n标签生成公式:")
print(f" future_return = (未来价格 - 当前价格) / 当前价格")
print(f" label = 1 if future_return > 0 else 0")
print(f"\n具体例子(前10个样本):")
print(f"{'时间':<20} {'当前价格':<12} {'未来价格':<12} {'收益率':<10} {'标签':<8}")
print("-"*80)
for i in range(10):
row = df_sample.iloc[i]
if pd.notna(row['future_return']):
label = "上涨(1)" if row['label_classification'] == 1 else "下跌(0)"
print(f"{str(row['timestamp']):<20} ${row['close']:<11,.2f} ${row['future_price']:<11,.2f} "
f"{row['future_return']*100:>7.2f}% {label:<8}")
print(f"\n标签统计:")
n_up = df_sample['label_classification'].sum()
n_down = len(df_sample) - n_up - df_sample['label_classification'].isna().sum()
print(f" 上涨(1): {n_up} 个")
print(f" 下跌(0): {n_down} 个")
print(f" 比例: {n_up/(n_up+n_down)*100:.1f}% vs {n_down/(n_up+n_down)*100:.1f}%")
print(f"\n模型输出:")
print(f" 格式: 0-1之间的概率值")
print(f" 例如: 0.73 表示 73% 概率上涨")
print(f" 决策: 如果概率 > 0.5 → 预测上涨")
print(f" 如果概率 < 0.5 → 预测下跌")
print(f"\n评估指标:")
print(f" - 准确率 (Accuracy)")
print(f" - 精确率 (Precision)")
print(f" - 召回率 (Recall)")
print(f" - F1分数")
print(f" - 混淆矩阵")
print(f"\n优点:")
print(f" ✅ 简单直观(涨/跌二选一)")
print(f" ✅ 评估指标清晰")
print(f" ✅ 适合交易决策(做多/做空)")
print(f"\n缺点:")
print(f" ❌ 丢失了涨跌幅度信息")
print(f" ❌ 涨0.1%和涨10%被视为一样")
print(f" ❌ 无法评估风险收益比")
# 回归任务
print("\n" + "="*80)
print("📈 任务2: 回归(Regression)")
print("="*80)
df_sample['label_regression'] = df_sample['future_return']
print(f"\n目标: 预测价格变化的百分比")
print(f"\n标签生成公式:")
print(f" label = (未来价格 - 当前价格) / 当前价格")
print(f" 直接预测收益率(可正可负)")
print(f"\n具体例子(前10个样本):")
print(f"{'时间':<20} {'当前价格':<12} {'未来价格':<12} {'收益率(标签)':<15}")
print("-"*80)
for i in range(10):
row = df_sample.iloc[i]
if pd.notna(row['future_return']):
print(f"{str(row['timestamp']):<20} ${row['close']:<11,.2f} ${row['future_price']:<11,.2f} "
f"{row['label_regression']*100:>12.4f}%")
print(f"\n标签统计:")
valid_returns = df_sample['label_regression'].dropna()
print(f" 最小值: {valid_returns.min()*100:.4f}%")
print(f" 最大值: {valid_returns.max()*100:.4f}%")
print(f" 均值: {valid_returns.mean()*100:.4f}%")
print(f" 标准差: {valid_returns.std()*100:.4f}%")
print(f" 中位数: {valid_returns.median()*100:.4f}%")
print(f"\n模型输出:")
print(f" 格式: 连续的数值(收益率)")
print(f" 例如: 0.0235 表示预测上涨 2.35%")
print(f" -0.0178 表示预测下跌 1.78%")
print(f" 决策: 根据预测值大小决定仓位")
print(f" 预测+5% → 大仓位做多")
print(f" 预测+0.5% → 小仓位做多")
print(f"\n评估指标:")
print(f" - RMSE (均方根误差)")
print(f" - MAE (平均绝对误差)")
print(f" - MAPE (平均绝对百分比误差)")
print(f" - 方向准确率 (预测方向是否正确)")
print(f" - R² (决定系数)")
print(f"\n优点:")
print(f" ✅ 保留了涨跌幅度信息")
print(f" ✅ 可以评估风险收益比")
print(f" ✅ 支持动态仓位管理")
print(f" ✅ 提供更精细的预测")
print(f"\n缺点:")
print(f" ❌ 预测难度更高")
print(f" ❌ 容易过拟合")
print(f" ❌ 对异常值敏感")
# 对比分析
print("\n[3/3] 详细对比分析...")
print("\n" + "="*80)
print("🔍 详细对比")
print("="*80)
comparison = [
["维度", "二分类", "回归"],
["-"*30, "-"*40, "-"*40],
["预测目标", "方向(涨/跌)", "具体涨跌幅度"],
["输出范围", "0-1(概率)", "-∞ to +∞(收益率)"],
["输出示例", "0.73(73%概率上涨)", "+0.0235(预测涨2.35%)"],
["标签类型", "离散(0或1)", "连续(实数)"],
["损失函数", "BCEWithLogitsLoss", "MSELoss / MAELoss"],
["评估指标", "准确率、F1分数", "RMSE、MAE、MAPE"],
["决策方式", "二元(做多/做空)", "连续(仓位大小)"],
["信息量", "少(只有方向)", "多(方向+幅度)"],
["预测难度", "中等", "困难"],
["适用场景", "简单交易策略", "精细化资金管理"],
]
for row in comparison:
print(f"{row[0]:<30} {row[1]:<40} {row[2]:<40}")
# 实际应用对比
print("\n" + "="*80)
print("💰 实际交易应用对比")
print("="*80)
print("\n场景1: 预测未来1小时BTC会上涨")
print("\n二分类模型:")
print(" 输出: 0.68 (68%概率上涨)")
print(" 决策: 概率>0.5,买入BTC")
print(" 行动: 全仓做多(或固定仓位如50%)")
print(" 问题: 不知道预期涨多少,无法评估风险")
print("\n回归模型:")
print(" 输出: +0.0023 (预测上涨0.23%)")
print(" 决策: 预测值>0,买入BTC")
print(" 行动: 根据预测幅度调整仓位(0.23%很小,可能只用10%仓位)")
print(" 优势: 知道预期收益,可以计算风险收益比")
print("\n场景2: 预测未来1小时BTC大涨")
print("\n二分类模型:")
print(" 输出: 0.72 (72%概率上涨)")
print(" 决策: 买入BTC")
print(" 问题: 72%概率可能是涨0.1%,也可能是涨5%,无法区分")
print("\n回归模型:")
print(" 输出: +0.0480 (预测上涨4.8%)")
print(" 决策: 买入BTC")
print(" 行动: 预期涨幅大,可以加大仓位(如80%)")
print(" 优势: 可以根据预期收益调整风险敞口")
# 性能对比
print("\n" + "="*80)
print("📊 当前项目的实际性能对比")
print("="*80)
print("\n二分类任务(当前已实现):")
print(" 准确率: 49.76%(接近随机)")
print(" F1分数: 0.3898")
print(" 召回率: 31.85%")
print(" 问题: 预测能力极弱")
print("\n回归任务(未实现):")
print(" 预期性能: 可能更差")
print(" 原因: 预测具体数值比预测方向更难")
print(" 但优势: 即使方向准确率50%,如果能准确预测幅度,仍可盈利")
# 建议
print("\n" + "="*80)
print("💡 建议")
print("="*80)
print("\n选择二分类的情况:")
print(" ✅ 初学者或概念验证")
print(" ✅ 简单的做多/做空策略")
print(" ✅ 预测周期较长(1天以上)")
print(" ✅ 只关心方向,不关心幅度")
print("\n选择回归的情况:")
print(" ✅ 需要精细化资金管理")
print(" ✅ 需要评估风险收益比")
print(" ✅ 有足够多的训练数据")
print(" ✅ 想要更丰富的预测信息")
print("\n对于1小时BTC预测:")
print(" 当前结论: 两者都很难!")
print(" 二分类: 准确率49.76%(已验证)")
print(" 回归: 预计MAPE会很高(未验证,但大概率也很差)")
print("\n 核心问题: 1小时价格变化接近随机游走")
print(" 建议: 延长预测周期到24小时或更长")
print("\n" + "="*80)
print("✓ 对比分析完成")
print("="*80)
if __name__ == "__main__":
compare_classification_vs_regression()
+1
View File
@@ -0,0 +1 @@
"""Data fetching, preprocessing, and dataset utilities."""
+147
View File
@@ -0,0 +1,147 @@
"""PyTorch Dataset classes for cryptocurrency time series data."""
from typing import Literal, Tuple
import numpy as np
import torch
from torch.utils.data import Dataset
class CryptoDataset(Dataset):
"""PyTorch Dataset for cryptocurrency time series prediction."""
def __init__(
self,
X: np.ndarray,
y: np.ndarray,
task: Literal['classification', 'regression'] = 'classification'
):
"""Initialize dataset.
Args:
X: Feature sequences of shape (n_samples, lookback, n_features)
y: Labels of shape (n_samples,)
task: Task type ('classification' or 'regression')
"""
self.X = torch.FloatTensor(X)
if task == 'classification':
self.y = torch.LongTensor(y)
else: # regression
self.y = torch.FloatTensor(y)
self.task = task
def __len__(self) -> int:
"""Return number of samples."""
return len(self.X)
def __getitem__(self, idx: int) -> Tuple[torch.Tensor, torch.Tensor]:
"""Get a sample.
Args:
idx: Sample index
Returns:
Tuple of (features, label)
"""
return self.X[idx], self.y[idx]
def get_feature_dim(self) -> int:
"""Get number of features.
Returns:
Number of features
"""
return self.X.shape[-1]
def get_sequence_length(self) -> int:
"""Get sequence length (lookback).
Returns:
Sequence length
"""
return self.X.shape[1]
def create_dataloaders(
train_data: Tuple[np.ndarray, np.ndarray],
val_data: Tuple[np.ndarray, np.ndarray],
test_data: Tuple[np.ndarray, np.ndarray],
task: Literal['classification', 'regression'] = 'classification',
batch_size: int = 64,
num_workers: int = 0, # 0 for M1 compatibility
pin_memory: bool = False # False for MPS
) -> Tuple[torch.utils.data.DataLoader, ...]:
"""Create DataLoaders for train/val/test sets.
Args:
train_data: Tuple of (X_train, y_train)
val_data: Tuple of (X_val, y_val)
test_data: Tuple of (X_test, y_test)
task: Task type
batch_size: Batch size
num_workers: Number of worker processes (0 for M1)
pin_memory: Whether to pin memory (False for MPS)
Returns:
Tuple of (train_loader, val_loader, test_loader)
"""
# Create datasets
train_dataset = CryptoDataset(*train_data, task=task)
val_dataset = CryptoDataset(*val_data, task=task)
test_dataset = CryptoDataset(*test_data, task=task)
# Create dataloaders
train_loader = torch.utils.data.DataLoader(
train_dataset,
batch_size=batch_size,
shuffle=True,
num_workers=num_workers,
pin_memory=pin_memory
)
val_loader = torch.utils.data.DataLoader(
val_dataset,
batch_size=batch_size,
shuffle=False,
num_workers=num_workers,
pin_memory=pin_memory
)
test_loader = torch.utils.data.DataLoader(
test_dataset,
batch_size=batch_size,
shuffle=False,
num_workers=num_workers,
pin_memory=pin_memory
)
return train_loader, val_loader, test_loader
if __name__ == "__main__":
# Test the dataset
from fetcher import BinanceDataFetcher
from preprocessor import CryptoPreprocessor, split_data
# Fetch and preprocess data
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h')
preprocessor = CryptoPreprocessor(lookback=168)
X, y, _ = preprocessor.preprocess_pipeline(df, task='classification')
# Split data
train_data, val_data, test_data = split_data(X, y)
# Create dataloaders
train_loader, val_loader, test_loader = create_dataloaders(
train_data, val_data, test_data,
batch_size=32
)
# Test iteration
for batch_X, batch_y in train_loader:
print(f"Batch X shape: {batch_X.shape}")
print(f"Batch y shape: {batch_y.shape}")
break
+191
View File
@@ -0,0 +1,191 @@
"""从Binance获取加密货币OHLCV数据"""
import os
from datetime import datetime, timedelta
from pathlib import Path
from typing import Optional, Literal
import pandas as pd
from binance.client import Client
import pytz
from dotenv import load_dotenv
# 加载环境变量
load_dotenv()
class BinanceDataFetcher:
"""从Binance公开API获取和缓存OHLCV数据"""
INTERVALS = {
'1h': Client.KLINE_INTERVAL_1HOUR,
'15m': Client.KLINE_INTERVAL_15MINUTE,
}
def __init__(self, cache_dir: str = "./data/cache"):
"""初始化Binance客户端和缓存目录
Args:
cache_dir: 缓存下载数据的目录
"""
# 配置代理
proxies = None
use_proxy = os.getenv('USE_PROXY', 'false').lower() == 'true'
if use_proxy:
socks5_proxy = os.getenv('SOCKS5_PROXY', 'socks5://127.0.0.1:7890')
proxies = {
'http': socks5_proxy,
'https': socks5_proxy
}
print(f"✓ 使用SOCKS5代理: {socks5_proxy}")
try:
# 初始化Binance客户端(带代理)
if proxies:
self.client = Client(requests_params={'proxies': proxies})
else:
self.client = Client()
# 测试连接
self.client.ping()
print("✓ 已成功连接到Binance API")
except Exception as e:
print(f"⚠ Binance API连接失败: {e}")
print("提示:")
print(" 1. 检查代理设置 (.env文件)")
print(" 2. 确保代理服务正在运行")
print(" 3. 或者使用已缓存的数据")
self.client = None
self.cache_dir = Path(cache_dir)
self.cache_dir.mkdir(parents=True, exist_ok=True)
def fetch_ohlcv(
self,
symbol: Literal['BTC', 'ETH', 'SOL'],
timeframe: Literal['1h', '15m'],
start_date: Optional[str] = None,
end_date: Optional[str] = None,
use_cache: bool = True
) -> pd.DataFrame:
"""获取加密货币对的OHLCV数据
Args:
symbol: 加密货币符号 (BTC, ETH, SOL)
timeframe: 时间框架间隔 (1h 或 15m)
start_date: 开始日期,格式:'YYYY-MM-DD' (默认: 1年前)
end_date: 结束日期,格式:'YYYY-MM-DD' (默认: 现在)
use_cache: 是否使用缓存数据
Returns:
包含以下列的DataFrame: timestamp, open, high, low, close, volume
"""
# 生成缓存文件名
cache_file = self._get_cache_filename(symbol, timeframe, start_date, end_date)
# 检查缓存
if use_cache and cache_file.exists():
print(f"从缓存加载数据:{cache_file}")
return pd.read_csv(cache_file, parse_dates=['timestamp'])
# 检查客户端是否可用
if self.client is None:
raise ConnectionError(
"Binance客户端不可用。请使用VPN或使用已缓存的数据。"
)
# 设置默认日期
if end_date is None:
end_dt = datetime.now(pytz.UTC)
else:
end_dt = datetime.strptime(end_date, '%Y-%m-%d').replace(tzinfo=pytz.UTC)
if start_date is None:
start_dt = end_dt - timedelta(days=365) # 1年前
else:
start_dt = datetime.strptime(start_date, '%Y-%m-%d').replace(tzinfo=pytz.UTC)
# Binance使用USDT交易对
pair = f"{symbol}USDT"
interval = self.INTERVALS[timeframe]
print(f"获取 {symbol} {timeframe} 数据,时间范围:{start_dt.date()} 至 {end_dt.date()}...")
# 获取K线数据
klines = self.client.get_historical_klines(
symbol=pair,
interval=interval,
start_str=str(int(start_dt.timestamp() * 1000)),
end_str=str(int(end_dt.timestamp() * 1000))
)
if not klines:
raise ValueError(f"未获取到 {symbol} {timeframe} 的数据")
# 转换为DataFrame
df = pd.DataFrame(klines, columns=[
'timestamp', 'open', 'high', 'low', 'close', 'volume',
'close_time', 'quote_volume', 'trades', 'taker_buy_base',
'taker_buy_quote', 'ignore'
])
# 选择并转换列
df = df[['timestamp', 'open', 'high', 'low', 'close', 'volume']].copy()
df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
for col in ['open', 'high', 'low', 'close', 'volume']:
df[col] = df[col].astype(float)
# 按时间排序
df = df.sort_values('timestamp').reset_index(drop=True)
print(f"已获取 {len(df)} 行数据,时间范围:{df['timestamp'].min()} 至 {df['timestamp'].max()}")
# 保存到缓存
df.to_csv(cache_file, index=False)
print(f"已缓存数据至:{cache_file}")
return df
def _get_cache_filename(
self,
symbol: str,
timeframe: str,
start_date: Optional[str],
end_date: Optional[str]
) -> Path:
"""根据参数生成缓存文件名"""
start = start_date or "1y"
end = end_date or "now"
filename = f"{symbol}_{timeframe}_{start}_{end}.csv"
return self.cache_dir / filename
def get_latest_price(self, symbol: Literal['BTC', 'ETH', 'SOL']) -> float:
"""获取符号的当前价格
Args:
symbol: 加密货币符号
Returns:
当前USDT价格
"""
pair = f"{symbol}USDT"
ticker = self.client.get_symbol_ticker(symbol=pair)
return float(ticker['price'])
if __name__ == "__main__":
# 测试数据获取器
fetcher = BinanceDataFetcher()
# 获取BTC 1小时数据(1年)
df = fetcher.fetch_ohlcv('BTC', '1h')
print("\nBTC 1小时数据:")
print(df.head())
print(f"\n形状:{df.shape}")
print(f"日期范围:{df['timestamp'].min()} 至 {df['timestamp'].max()}")
# 获取当前价格
current_price = fetcher.get_latest_price('BTC')
print(f"\n当前BTC价格:${current_price:,.2f}")
+329
View File
@@ -0,0 +1,329 @@
"""Data preprocessing: normalization, feature engineering, and sequence creation."""
from typing import Tuple, Optional, Literal
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
import pandas_ta as ta
class CryptoPreprocessor:
"""Preprocess cryptocurrency data with normalization and feature engineering."""
def __init__(
self,
lookback: int = 168,
forecast_horizon: int = 24,
price_change_threshold: float = 0.0 # 改为0,使用真实涨跌
):
"""Initialize preprocessor.
Args:
lookback: Number of historical timesteps to use
forecast_horizon: Number of timesteps to predict ahead
price_change_threshold: Threshold for binary classification (e.g., 0.015 = 1.5%)
"""
self.lookback = lookback
self.forecast_horizon = forecast_horizon
self.price_change_threshold = price_change_threshold
self.scaler = MinMaxScaler()
self.feature_columns = []
self.is_fitted = False
def add_technical_indicators(self, df: pd.DataFrame) -> pd.DataFrame:
"""添加技术指标作为特征
Args:
df: 包含OHLCV数据的DataFrame
Returns:
添加了技术指标的DataFrame
"""
df = df.copy()
# 使用pandas_ta库
# RSI (14 periods)
df['rsi'] = ta.rsi(df['close'], length=14)
# MACD
macd = ta.macd(df['close'])
if macd is not None and not macd.empty:
# pandas-ta 返回的列名可能不同,动态获取
macd_cols = macd.columns.tolist()
if len(macd_cols) >= 3:
df['macd'] = macd.iloc[:, 0] # MACD线
df['macd_hist'] = macd.iloc[:, 1] # MACD柱
df['macd_signal'] = macd.iloc[:, 2] # 信号线
else:
# 如果格式不对,使用None
df['macd'] = None
df['macd_signal'] = None
df['macd_hist'] = None
else:
df['macd'] = None
df['macd_signal'] = None
df['macd_hist'] = None
# Bollinger Bands
bbands = ta.bbands(df['close'], length=20)
if bbands is not None and not bbands.empty:
# pandas-ta 返回的列名可能不同,动态获取
bb_cols = bbands.columns.tolist()
if len(bb_cols) >= 3:
df['bb_lower'] = bbands.iloc[:, 0] # 第一列是下轨
df['bb_middle'] = bbands.iloc[:, 1] # 第二列是中轨
df['bb_upper'] = bbands.iloc[:, 2] # 第三列是上轨
df['bb_bandwidth'] = (df['bb_upper'] - df['bb_lower']) / (df['bb_middle'] + 1e-8)
else:
# 如果格式不对,手动计算
sma = df['close'].rolling(window=20).mean()
std = df['close'].rolling(window=20).std()
df['bb_middle'] = sma
df['bb_upper'] = sma + (std * 2)
df['bb_lower'] = sma - (std * 2)
df['bb_bandwidth'] = (df['bb_upper'] - df['bb_lower']) / (df['bb_middle'] + 1e-8)
else:
# pandas-ta不可用,手动计算布林带
sma = df['close'].rolling(window=20).mean()
std = df['close'].rolling(window=20).std()
df['bb_middle'] = sma
df['bb_upper'] = sma + (std * 2)
df['bb_lower'] = sma - (std * 2)
df['bb_bandwidth'] = (df['bb_upper'] - df['bb_lower']) / (df['bb_middle'] + 1e-8)
# Volume indicators
df['volume_sma'] = ta.sma(df['volume'], length=20)
df['volume_ratio'] = df['volume'] / (df['volume_sma'] + 1e-8)
# Price momentum
df['returns'] = df['close'].pct_change()
df['returns_5'] = df['close'].pct_change(5)
df['returns_10'] = df['close'].pct_change(10)
# Volatility
df['volatility'] = df['returns'].rolling(window=20).std()
return df
def add_time_features(self, df: pd.DataFrame) -> pd.DataFrame:
"""Add time-based features.
Args:
df: DataFrame with timestamp column
Returns:
DataFrame with added time features
"""
df = df.copy()
df['hour'] = df['timestamp'].dt.hour
df['day_of_week'] = df['timestamp'].dt.dayofweek
df['is_weekend'] = (df['day_of_week'] >= 5).astype(int)
# Cyclical encoding for hour
df['hour_sin'] = np.sin(2 * np.pi * df['hour'] / 24)
df['hour_cos'] = np.cos(2 * np.pi * df['hour'] / 24)
# Cyclical encoding for day of week
df['dow_sin'] = np.sin(2 * np.pi * df['day_of_week'] / 7)
df['dow_cos'] = np.cos(2 * np.pi * df['day_of_week'] / 7)
return df
def create_labels(
self,
df: pd.DataFrame,
task: Literal['classification', 'regression']
) -> pd.DataFrame:
"""Create target labels for classification or regression.
Args:
df: DataFrame with price data
task: Task type ('classification' or 'regression')
Returns:
DataFrame with added label column
"""
df = df.copy()
# Calculate future returns
df['future_price'] = df['close'].shift(-self.forecast_horizon)
df['future_return'] = (df['future_price'] - df['close']) / df['close']
if task == 'classification':
# Binary classification: 1 if price increases above threshold, 0 otherwise
df['label'] = (df['future_return'] > self.price_change_threshold).astype(int)
else: # regression
# Regression: predict percentage return
df['label'] = df['future_return']
# Drop rows without labels (last forecast_horizon rows)
df = df[:-self.forecast_horizon].copy()
return df
def fit(self, df: pd.DataFrame) -> 'CryptoPreprocessor':
"""Fit the scaler on training data.
Args:
df: Training DataFrame with features
Returns:
Self
"""
# Get feature columns (exclude timestamp and label)
exclude_cols = ['timestamp', 'label', 'future_price', 'future_return']
self.feature_columns = [col for col in df.columns if col not in exclude_cols]
# Fit scaler
self.scaler.fit(df[self.feature_columns].fillna(0))
self.is_fitted = True
return self
def transform(self, df: pd.DataFrame) -> np.ndarray:
"""Transform data using fitted scaler.
Args:
df: DataFrame with features
Returns:
Scaled feature array
"""
if not self.is_fitted:
raise ValueError("Preprocessor must be fitted before transform")
# Fill NaN values
df_filled = df[self.feature_columns].fillna(0)
# Scale features
scaled = self.scaler.transform(df_filled)
return scaled
def fit_transform(self, df: pd.DataFrame) -> np.ndarray:
"""Fit and transform in one step.
Args:
df: DataFrame with features
Returns:
Scaled feature array
"""
return self.fit(df).transform(df)
def create_sequences(
self,
features: np.ndarray,
labels: np.ndarray
) -> Tuple[np.ndarray, np.ndarray]:
"""Create sequences for time series prediction.
Args:
features: Feature array (n_samples, n_features)
labels: Label array (n_samples,)
Returns:
Tuple of (X, y) where X is (n_sequences, lookback, n_features)
and y is (n_sequences,)
"""
X, y = [], []
for i in range(len(features) - self.lookback):
X.append(features[i:i + self.lookback])
y.append(labels[i + self.lookback])
return np.array(X), np.array(y)
def preprocess_pipeline(
self,
df: pd.DataFrame,
task: Literal['classification', 'regression'],
fit: bool = True
) -> Tuple[np.ndarray, np.ndarray, pd.DataFrame]:
"""Full preprocessing pipeline.
Args:
df: Raw OHLCV DataFrame
task: Task type
fit: Whether to fit the scaler (True for training data)
Returns:
Tuple of (X, y, processed_df) where X is sequences, y is labels,
and processed_df is the DataFrame after feature engineering
"""
# Add features
df = self.add_technical_indicators(df)
df = self.add_time_features(df)
df = self.create_labels(df, task)
# Drop initial NaN rows from indicators
df = df.dropna().reset_index(drop=True)
# Scale features
if fit:
features = self.fit_transform(df)
else:
features = self.transform(df)
# Get labels
labels = df['label'].values
# Create sequences
X, y = self.create_sequences(features, labels)
print(f"Created {len(X)} sequences with shape {X.shape}")
return X, y, df
def split_data(
X: np.ndarray,
y: np.ndarray,
train_split: float = 0.7,
val_split: float = 0.15
) -> Tuple[Tuple[np.ndarray, np.ndarray], ...]:
"""Split data into train/val/test sets using time-based split.
Args:
X: Feature sequences
y: Labels
train_split: Fraction for training
val_split: Fraction for validation
Returns:
Tuple of ((X_train, y_train), (X_val, y_val), (X_test, y_test))
"""
n = len(X)
train_end = int(n * train_split)
val_end = int(n * (train_split + val_split))
X_train, y_train = X[:train_end], y[:train_end]
X_val, y_val = X[train_end:val_end], y[train_end:val_end]
X_test, y_test = X[val_end:], y[val_end:]
print(f"Train: {len(X_train)} samples")
print(f"Val: {len(X_val)} samples")
print(f"Test: {len(X_test)} samples")
return (X_train, y_train), (X_val, y_val), (X_test, y_test)
if __name__ == "__main__":
# Test the preprocessor
from fetcher import BinanceDataFetcher
# Fetch data
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h')
# Preprocess for classification
preprocessor = CryptoPreprocessor(lookback=168, forecast_horizon=24)
X, y, processed_df = preprocessor.preprocess_pipeline(df, task='classification')
print(f"\nFeatures shape: {X.shape}")
print(f"Labels shape: {y.shape}")
print(f"Label distribution: {np.bincount(y.astype(int))}")
# Split data
(X_train, y_train), (X_val, y_val), (X_test, y_test) = split_data(X, y)
+234
View File
@@ -0,0 +1,234 @@
"""诊断模型为何倾向预测上涨"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import torch
import numpy as np
import pandas as pd
from models.dlinear.model import DLinear
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
def analyze_bias():
print("="*80)
print("分析模型预测偏差的原因")
print("="*80)
# 1. 加载数据
print("\n[1/5] 加载数据并分析标签分布...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h', use_cache=True)
# 分析原始价格趋势
df['price_change'] = df['close'].pct_change()
df['future_1h'] = df['close'].shift(-1)
df['future_return_1h'] = (df['future_1h'] - df['close']) / df['close']
print(f"\n📊 原始数据分析(最近8760小时):")
print(f" 数据范围: {df['timestamp'].min()} 至 {df['timestamp'].max()}")
print(f" BTC起始价格: ${df['close'].iloc[0]:,.2f}")
print(f" BTC结束价格: ${df['close'].iloc[-1]:,.2f}")
total_return = (df['close'].iloc[-1] - df['close'].iloc[0]) / df['close'].iloc[0]
print(f" 总收益率: {total_return*100:.2f}%")
up_hours = (df['future_return_1h'] > 0).sum()
down_hours = (df['future_return_1h'] <= 0).sum()
total_hours = up_hours + down_hours
print(f"\n 小时K线涨跌统计:")
print(f" 上涨小时: {up_hours}/{total_hours} ({up_hours/total_hours*100:.1f}%)")
print(f" 下跌小时: {down_hours}/{total_hours} ({down_hours/total_hours*100:.1f}%)")
# 2. 预处理并查看训练集分布
print("\n[2/5] 分析训练/验证/测试集的标签分布...")
preprocessor = CryptoPreprocessor(
lookback=168,
forecast_horizon=1,
price_change_threshold=0.0
)
X, y, processed_df = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
train_data, val_data, test_data = split_data(X, y, train_split=0.7, val_split=0.15)
# 分析每个数据集的标签分布
train_up = (train_data[1] == 1).sum()
train_down = (train_data[1] == 0).sum()
val_up = (val_data[1] == 1).sum()
val_down = (val_data[1] == 0).sum()
test_up = (test_data[1] == 1).sum()
test_down = (test_data[1] == 0).sum()
print(f"\n 训练集标签分布:")
print(f" 上涨: {train_up}/{len(train_data[1])} ({train_up/len(train_data[1])*100:.1f}%)")
print(f" 下跌: {train_down}/{len(train_data[1])} ({train_down/len(train_data[1])*100:.1f}%)")
print(f"\n 验证集标签分布:")
print(f" 上涨: {val_up}/{len(val_data[1])} ({val_up/len(val_data[1])*100:.1f}%)")
print(f" 下跌: {val_down}/{len(val_data[1])} ({val_down/len(val_data[1])*100:.1f}%)")
print(f"\n 测试集标签分布:")
print(f" 上涨: {test_up}/{len(test_data[1])} ({test_up/len(test_data[1])*100:.1f}%)")
print(f" 下跌: {test_down}/{len(test_data[1])} ({test_down/len(test_data[1])*100:.1f}%)")
# 3. 分析训练集对应的时间范围
print("\n[3/5] 分析各数据集的时间范围和价格趋势...")
# 由于序列创建,需要偏移lookback
lookback = 168
train_end_idx = int(len(processed_df) * 0.7) - lookback
val_end_idx = int(len(processed_df) * 0.85) - lookback
train_df = processed_df.iloc[lookback:train_end_idx+lookback]
val_df = processed_df.iloc[train_end_idx+lookback:val_end_idx+lookback]
test_df = processed_df.iloc[val_end_idx+lookback:]
print(f"\n 训练集:")
print(f" 时间范围: {train_df['timestamp'].min()} 至 {train_df['timestamp'].max()}")
train_return = (train_df['close'].iloc[-1] - train_df['close'].iloc[0]) / train_df['close'].iloc[0]
print(f" 价格变化: ${train_df['close'].iloc[0]:,.0f} → ${train_df['close'].iloc[-1]:,.0f}")
print(f" 收益率: {train_return*100:.2f}%")
print(f"\n 验证集:")
print(f" 时间范围: {val_df['timestamp'].min()} 至 {val_df['timestamp'].max()}")
val_return = (val_df['close'].iloc[-1] - val_df['close'].iloc[0]) / val_df['close'].iloc[0]
print(f" 价格变化: ${val_df['close'].iloc[0]:,.0f} → ${val_df['close'].iloc[-1]:,.0f}")
print(f" 收益率: {val_return*100:.2f}%")
print(f"\n 测试集:")
print(f" 时间范围: {test_df['timestamp'].min()} 至 {test_df['timestamp'].max()}")
test_return = (test_df['close'].iloc[-1] - test_df['close'].iloc[0]) / test_df['close'].iloc[0]
print(f" 价格变化: ${test_df['close'].iloc[0]:,.0f} → ${test_df['close'].iloc[-1]:,.0f}")
print(f" 收益率: {test_return*100:.2f}%")
# 4. 加载模型并分析输出
print("\n[4/5] 加载模型并分析输出分布...")
model_path = './models/checkpoints/dlinear_btc_1h_forecast/best_model.pt'
device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu')
checkpoint = torch.load(model_path, map_location=device, weights_only=False)
model = DLinear(checkpoint['config'])
model.load_state_dict(checkpoint['model_state_dict'])
model = model.to(device)
model.eval()
# 分析测试集输出
_, _, test_loader = create_dataloaders(
test_data, test_data, test_data,
task='classification',
batch_size=64
)
all_logits = []
all_labels = []
with torch.no_grad():
for batch_X, batch_y in test_loader:
batch_X = batch_X.to(device)
outputs = model(batch_X).squeeze()
all_logits.extend(outputs.cpu().numpy().flatten())
all_labels.extend(batch_y.numpy().flatten())
all_logits = np.array(all_logits)
all_labels = np.array(all_labels)
all_probs = 1 / (1 + np.exp(-all_logits)) # sigmoid
print(f"\n 模型输出统计:")
print(f" Logits均值: {all_logits.mean():.6f}")
print(f" Logits标准差: {all_logits.std():.6f}")
print(f" Logits中位数: {np.median(all_logits):.6f}")
print(f" 概率均值: {all_probs.mean():.6f}")
# 按真实标签分组分析
logits_when_up = all_logits[all_labels == 1]
logits_when_down = all_logits[all_labels == 0]
print(f"\n 按真实标签分组:")
print(f" 真实上涨时,平均logit: {logits_when_up.mean():.6f}")
print(f" 真实下跌时,平均logit: {logits_when_down.mean():.6f}")
print(f" 差值: {logits_when_up.mean() - logits_when_down.mean():.6f}")
# 5. 分析为什么偏向上涨
print("\n[5/5] 分析预测偏差的根本原因...")
print("\n" + "="*80)
print("🔍 根本原因分析")
print("="*80)
# 原因1: 数据趋势
if total_return > 0.1:
print(f"\n✅ 原因1: 训练数据处于上涨趋势")
print(f" - 整体收益率: {total_return*100:.2f}%")
print(f" - 训练集收益率: {train_return*100:.2f}%")
print(f" - 模型学习到了'牛市偏好'")
elif total_return < -0.1:
print(f"\n⚠️ 原因1: 训练数据处于下跌趋势")
print(f" - 整体收益率: {total_return*100:.2f}%")
else:
print(f"\n⚪ 原因1: 训练数据趋势中性")
print(f" - 整体收益率: {total_return*100:.2f}%")
# 原因2: 类别不平衡
if train_up > train_down * 1.05:
print(f"\n✅ 原因2: 训练集标签不平衡")
print(f" - 上涨样本: {train_up} ({train_up/len(train_data[1])*100:.1f}%)")
print(f" - 下跌样本: {train_down} ({train_down/len(train_data[1])*100:.1f}%)")
print(f" - 不平衡比例: {train_up/train_down:.2f}:1")
else:
print(f"\n⚪ 原因2: 训练集标签基本平衡")
print(f" - 上涨/下跌比例: {train_up/train_down:.2f}:1")
# 原因3: 模型输出偏差
if all_logits.mean() > 0.01:
print(f"\n✅ 原因3: 模型输出存在正偏差")
print(f" - 平均logit: {all_logits.mean():.6f} > 0")
print(f" - 平均概率: {all_probs.mean():.6f} > 0.5")
print(f" - 这导致更多样本被预测为上涨")
elif all_logits.mean() < -0.01:
print(f"\n⚠️ 原因3: 模型输出存在负偏差")
print(f" - 平均logit: {all_logits.mean():.6f} < 0")
else:
print(f"\n⚪ 原因3: 模型输出基本无偏")
print(f" - 平均logit: {all_logits.mean():.6f} ≈ 0")
# 原因4: 阈值问题
print(f"\n⚪ 原因4: 决策阈值")
print(f" - 当前阈值: 0.5")
print(f" - 平均概率: {all_probs.mean():.6f}")
if all_probs.mean() > 0.5:
print(f" - 因为平均概率>0.5,导致多数样本被预测为上涨")
# 计算最优阈值
from sklearn.metrics import f1_score
best_threshold = 0.5
best_f1 = 0
for threshold in np.arange(0.3, 0.7, 0.01):
preds = (all_probs > threshold).astype(int)
f1 = f1_score(all_labels, preds)
if f1 > best_f1:
best_f1 = f1
best_threshold = threshold
print(f"\n💡 建议调整阈值:")
print(f" - 最优阈值: {best_threshold:.3f} (F1={best_f1:.4f})")
print(f" - 当前阈值: 0.500 (F1={f1_score(all_labels, (all_probs>0.5).astype(int)):.4f})")
print("\n" + "="*80)
print("📌 总结")
print("="*80)
print(f"\n模型预测上涨的比例: {(all_probs > 0.5).sum()}/{len(all_probs)} ({(all_probs>0.5).mean()*100:.1f}%)")
print(f"主要原因:")
if total_return > 0.1:
print(f" 1. ✅ 训练数据处于上涨趋势 ({total_return*100:.1f}%)")
if train_up > train_down * 1.05:
print(f" 2. ✅ 训练集上涨样本更多 ({train_up/train_down:.2f}:1)")
if all_logits.mean() > 0.01:
print(f" 3. ✅ 模型学习到了上涨偏好 (平均logit={all_logits.mean():.4f})")
print(f"\n这是正常现象,因为:")
print(f" - 如果训练期间BTC整体上涨,模型会学习到'上涨更可能'")
print(f" - 这类似于'在牛市训练的模型会更乐观'")
print(f" - 解决方案:使用更长的训练数据(包含牛市和熊市)")
if __name__ == "__main__":
analyze_bias()
+177
View File
@@ -0,0 +1,177 @@
"""诊断模型预测偏差问题"""
import torch
import numpy as np
from models.dlinear.model import DLinear, DLinearConfig
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
def diagnose_model():
"""诊断模型输出分布和损失函数问题"""
print("="*80)
print("模型诊断报告")
print("="*80)
# 1. 加载已训练的模型
print("\n[1/5] 加载已训练模型...")
model_path = './models/checkpoints/dlinear_btc_1h_balanced/best_model.pt'
device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu')
checkpoint = torch.load(model_path, map_location=device, weights_only=False)
config = checkpoint['config']
model = DLinear(config)
model.load_state_dict(checkpoint['model_state_dict'])
model = model.to(device)
model.eval()
print(f"✓ 模型已加载到设备: {device}")
print(f"✓ 模型配置: task={config.task}, input_dim={config.input_dim}")
# 2. 准备测试数据
print("\n[2/5] 准备测试数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h', use_cache=True)
preprocessor = CryptoPreprocessor(lookback=168, forecast_horizon=24, price_change_threshold=0.0)
X, y, _ = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
_, _, test_data = split_data(X, y, train_split=0.7, val_split=0.15)
_, _, test_loader = create_dataloaders(test_data, test_data, test_data, batch_size=64)
# 3. 分析模型输出
print("\n[3/5] 分析模型原始输出...")
all_outputs = []
all_labels = []
with torch.no_grad():
for batch_X, batch_y in test_loader:
batch_X = batch_X.to(device)
outputs = model(batch_X).squeeze()
all_outputs.extend(outputs.cpu().numpy().flatten())
all_labels.extend(batch_y.numpy().flatten())
all_outputs = np.array(all_outputs)
all_labels = np.array(all_labels)
print(f"\n📊 模型输出统计(经过Sigmoid):")
print(f" 最小值: {all_outputs.min():.6f}")
print(f" 最大值: {all_outputs.max():.6f}")
print(f" 均值: {all_outputs.mean():.6f}")
print(f" 中位数: {np.median(all_outputs):.6f}")
print(f" 标准差: {all_outputs.std():.6f}")
print(f" > 0.5 的比例: {(all_outputs > 0.5).mean()*100:.2f}%")
print(f" > 0.4 的比例: {(all_outputs > 0.4).mean()*100:.2f}%")
print(f" > 0.3 的比例: {(all_outputs > 0.3).mean()*100:.2f}%")
print(f" > 0.2 的比例: {(all_outputs > 0.2).mean()*100:.2f}%")
print(f" > 0.1 的比例: {(all_outputs > 0.1).mean()*100:.2f}%")
print(f"\n📊 真实标签分布:")
print(f" 下跌(0): {(all_labels == 0).sum()} ({(all_labels == 0).mean()*100:.1f}%)")
print(f" 上涨(1): {(all_labels == 1).sum()} ({(all_labels == 1).mean()*100:.1f}%)")
# 4. 检查损失函数问题
print("\n[4/5] 检查损失函数兼容性...")
print("\n⚠️ 发现问题:")
print(" 模型使用: nn.Sigmoid() 作为输出激活函数")
print(" 当前损失函数: BCEWithLogitsLoss (期望logits输入)")
print("\n🔴 这导致了【双重Sigmoid问题】:")
print(" 1. 模型输出层: output = sigmoid(logits)")
print(" 2. 损失函数内部: sigmoid(output) = sigmoid(sigmoid(logits))")
print(" 3. 结果: 输出被过度压缩到接近0,梯度消失")
# 验证双重sigmoid效果
print("\n🧪 验证双重Sigmoid效果:")
test_logits = np.array([-2, -1, 0, 1, 2])
single_sigmoid = 1 / (1 + np.exp(-test_logits))
double_sigmoid = 1 / (1 + np.exp(-single_sigmoid))
print(" 原始logits: ", test_logits)
print(" 单次sigmoid: ", np.round(single_sigmoid, 4))
print(" 双重sigmoid: ", np.round(double_sigmoid, 4))
print(" → 双重sigmoid后,所有值都接近0.5(失去区分度)")
# 5. 建议修复方案
print("\n[5/5] 修复方案:")
print("\n✅ 方案1(推荐):移除模型输出层的Sigmoid")
print(" - 修改 models/dlinear/model.py:128")
print(" - 将 self.output_activation = nn.Sigmoid() 改为 nn.Identity()")
print(" - 保持损失函数为 BCEWithLogitsLoss")
print("\n✅ 方案2:更换损失函数")
print(" - 修改 models/base.py:74")
print(" - 将 BCEWithLogitsLoss 改为 BCELoss")
print(" - 保持模型输出层的 Sigmoid")
print("\n📌 其他发现:")
print(f" - 模型平均输出 {all_outputs.mean():.1%},远低于期望的50%")
print(f" - 几乎没有输出>0.5({(all_outputs>0.5).sum()}个)")
print(f" - 这解释了为什么模型从不预测上涨")
# 6. 生成可视化
print("\n[6/5] 生成输出分布图...")
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
# 输出直方图
axes[0, 0].hist(all_outputs, bins=50, alpha=0.7, edgecolor='black')
axes[0, 0].axvline(0.5, color='red', linestyle='--', label='决策阈值=0.5')
axes[0, 0].set_xlabel('模型输出(经过Sigmoid)')
axes[0, 0].set_ylabel('频数')
axes[0, 0].set_title('模型输出分布(双重Sigmoid问题)')
axes[0, 0].legend()
axes[0, 0].grid(alpha=0.3)
# 按真实标签分组的输出分布
outputs_down = all_outputs[all_labels == 0]
outputs_up = all_outputs[all_labels == 1]
axes[0, 1].hist([outputs_down, outputs_up], bins=30, alpha=0.6,
label=['真实下跌', '真实上涨'], edgecolor='black')
axes[0, 1].axvline(0.5, color='red', linestyle='--', label='决策阈值')
axes[0, 1].set_xlabel('模型输出')
axes[0, 1].set_ylabel('频数')
axes[0, 1].set_title('按真实标签分组的输出分布')
axes[0, 1].legend()
axes[0, 1].grid(alpha=0.3)
# CDF(累积分布)
sorted_outputs = np.sort(all_outputs)
cdf = np.arange(1, len(sorted_outputs)+1) / len(sorted_outputs)
axes[1, 0].plot(sorted_outputs, cdf, linewidth=2)
axes[1, 0].axvline(0.5, color='red', linestyle='--', label='决策阈值=0.5')
axes[1, 0].axhline(0.5, color='green', linestyle='--', alpha=0.5)
axes[1, 0].set_xlabel('模型输出')
axes[1, 0].set_ylabel('累积概率')
axes[1, 0].set_title('输出累积分布函数(CDF)')
axes[1, 0].legend()
axes[1, 0].grid(alpha=0.3)
# 双重sigmoid效果演示
x = np.linspace(-5, 5, 100)
sigmoid_1 = 1 / (1 + np.exp(-x))
sigmoid_2 = 1 / (1 + np.exp(-sigmoid_1))
axes[1, 1].plot(x, sigmoid_1, label='单次Sigmoid', linewidth=2)
axes[1, 1].plot(x, sigmoid_2, label='双重Sigmoid', linewidth=2, linestyle='--')
axes[1, 1].axhline(0.5, color='red', linestyle=':', alpha=0.5, label='决策阈值')
axes[1, 1].set_xlabel('输入(logits)')
axes[1, 1].set_ylabel('输出')
axes[1, 1].set_title('双重Sigmoid效果对比')
axes[1, 1].legend()
axes[1, 1].grid(alpha=0.3)
plt.tight_layout()
output_path = 'model_diagnosis.png'
plt.savefig(output_path, dpi=150, bbox_inches='tight')
print(f"✓ 可视化已保存至: {output_path}")
print("\n" + "="*80)
print("诊断完成!")
print("="*80)
if __name__ == "__main__":
diagnose_model()
+134
View File
@@ -0,0 +1,134 @@
"""简化版模型诊断(无需matplotlib)"""
import torch
import numpy as np
from models.dlinear.model import DLinear, DLinearConfig
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
def diagnose():
print("="*80)
print("🔍 模型预测偏差诊断报告")
print("="*80)
# 加载模型
print("\n[1/4] 加载已训练模型...")
model_path = './models/checkpoints/dlinear_btc_1h_balanced/best_model.pt'
device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu')
checkpoint = torch.load(model_path, map_location=device, weights_only=False)
config = checkpoint['config']
model = DLinear(config)
model.load_state_dict(checkpoint['model_state_dict'])
model = model.to(device)
model.eval()
print(f"✓ 模型配置: task={config.task}, hidden_dim={config.hidden_dim}")
# 准备数据
print("\n[2/4] 加载测试数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h', use_cache=True)
preprocessor = CryptoPreprocessor(lookback=168, forecast_horizon=24, price_change_threshold=0.0)
X, y, _ = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
_, _, test_data = split_data(X, y, train_split=0.7, val_split=0.15)
_, _, test_loader = create_dataloaders(test_data, test_data, test_data, batch_size=64)
# 收集输出
print("\n[3/4] 分析模型输出...")
all_outputs = []
all_labels = []
with torch.no_grad():
for batch_X, batch_y in test_loader:
batch_X = batch_X.to(device)
outputs = model(batch_X).squeeze()
all_outputs.extend(outputs.cpu().numpy().flatten())
all_labels.extend(batch_y.numpy().flatten())
all_outputs = np.array(all_outputs)
all_labels = np.array(all_labels)
# 统计分析
print("\n" + "="*80)
print("📊 关键发现")
print("="*80)
print(f"\n1️⃣ 模型输出统计(已经过Sigmoid):")
print(f" 最小值: {all_outputs.min():.6f}")
print(f" 最大值: {all_outputs.max():.6f}")
print(f" 均值: {all_outputs.mean():.6f} ← 应该接近0.5")
print(f" 中位数: {np.median(all_outputs):.6f}")
print(f" 标准差: {all_outputs.std():.6f}")
print(f"\n2️⃣ 预测阈值分析:")
for threshold in [0.5, 0.4, 0.3, 0.2, 0.1]:
count = (all_outputs > threshold).sum()
pct = (all_outputs > threshold).mean() * 100
print(f" 输出 > {threshold}: {count:4d} 个 ({pct:5.2f}%)")
print(f"\n3️⃣ 真实标签分布:")
n_down = (all_labels == 0).sum()
n_up = (all_labels == 1).sum()
print(f" 下跌(0): {n_down} ({n_down/len(all_labels)*100:.1f}%)")
print(f" 上涨(1): {n_up} ({n_up/len(all_labels)*100:.1f}%)")
# 验证双重sigmoid问题
print("\n" + "="*80)
print("🔴 问题诊断:双重Sigmoid Bug")
print("="*80)
print("\n📋 当前架构:")
print(" 模型输出层: self.output_activation = nn.Sigmoid() (models/dlinear/model.py:128)")
print(" 损失函数: BCEWithLogitsLoss() (models/base.py:74)")
print("\n⚠️ 问题:")
print(" BCEWithLogitsLoss 内部包含 Sigmoid,期望输入是 logits(未激活)")
print(" 但模型已经输出了 sigmoid(logits)")
print(" → 损失函数再次应用: sigmoid(sigmoid(logits)) ← 双重激活!")
print("\n🧪 双重Sigmoid效果演示:")
test_logits = np.array([-3, -1, 0, 1, 3])
single = 1 / (1 + np.exp(-test_logits))
double = 1 / (1 + np.exp(-single))
print(f" 原始logits: {test_logits}")
print(f" 单次sigmoid: {np.round(single, 4)}")
print(f" 双重sigmoid: {np.round(double, 4)} ← 全部压缩到~0.5")
print("\n💥 后果:")
print(f" - 模型输出均值仅 {all_outputs.mean():.1%}(期望~50%)")
print(f" - {(all_outputs>0.5).sum()} 个样本输出>0.5(共{len(all_outputs)}个)")
print(f" - 模型永远预测类别0(下跌),因为没有输出超过阈值0.5")
print(f" - 梯度消失,模型无法有效学习")
# 解决方案
print("\n" + "="*80)
print("✅ 修复方案")
print("="*80)
print("\n方案1(推荐):移除模型的Sigmoid输出")
print(" 文件: models/dlinear/model.py")
print(" 位置: 第127-130行")
print(" 修改前: if config.task == 'classification':")
print(" self.output_activation = nn.Sigmoid()")
print(" 修改后: if config.task == 'classification':")
print(" self.output_activation = nn.Identity() # 输出logits")
print(" 说明: 保持损失函数为BCEWithLogitsLoss")
print("\n方案2:更换损失函数(不推荐)")
print(" 文件: models/base.py")
print(" 位置: 第70-76行")
print(" 修改: 将 BCEWithLogitsLoss 改为 BCELoss")
print(" 说明: 保持模型输出Sigmoid,但BCELoss数值稳定性较差")
print("\n" + "="*80)
print("📌 结论")
print("="*80)
print("✓ 预测偏差的根本原因是【双重Sigmoid Bug】")
print("✓ 与数据处理、类别平衡、特征工程无关")
print("✓ 修复后需要重新训练模型")
print("="*80)
if __name__ == "__main__":
diagnose()
+63
View File
@@ -0,0 +1,63 @@
name: crypto_pred
channels:
- pytorch
- conda-forge
- defaults
dependencies:
# Python
- python=3.10
# PyTorch (Mac M1 支持)
- pytorch
- torchvision
- torchaudio
# 数据处理
- pandas>=2.0.0
- numpy>=1.24.0
- scikit-learn>=1.3.0
# 开发工具
- jupyter
- ipython
- tqdm
# 可视化
- matplotlib
- seaborn
# 通过pip安装的包
- pip
- pip:
# Binance API
- python-binance>=1.0.19
# 环境变量
- python-dotenv>=1.0.0
# 时区处理
- pytz
# ONNX
- onnx>=1.14.0
- onnxruntime>=1.15.0
# 技术指标
- pandas-ta>=0.3.14b
# 模型库
- darts>=0.27.0
- u8darts[torch]>=0.27.0
- tsfm_public>=0.2.0
- gluonts[torch]>=0.14.0
# FastAPI
- fastapi>=0.104.0
- uvicorn[standard]>=0.24.0
- pydantic>=2.0.0
- python-multipart>=0.0.6
# 测试
- pytest>=7.4.0
- black>=23.0.0
- flake8>=6.0.0
+180
View File
@@ -0,0 +1,180 @@
"""重新评估已训练模型(使用修复后的evaluator)"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import torch
import numpy as np
from models.dlinear.model import DLinear
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
from training.evaluator import evaluate_model
import json
def main():
print("="*80)
print("重新评估模型(已修复Sigmoid Bug)")
print("="*80)
# 加载模型
print("\n[1/4] 加载已训练的模型...")
model_path = './models/checkpoints/dlinear_btc_1h_forecast/best_model.pt'
device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu')
checkpoint = torch.load(model_path, map_location=device, weights_only=False)
config = checkpoint['config']
model = DLinear(config)
model.load_state_dict(checkpoint['model_state_dict'])
model = model.to(device)
model.eval()
print(f"✓ 模型已加载: {model_path}")
print(f"✓ 设备: {device}")
print(f"✓ 模型参数: {model.count_parameters():,}")
# 准备测试数据
print("\n[2/4] 准备测试数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h', use_cache=True)
preprocessor = CryptoPreprocessor(
lookback=168,
forecast_horizon=1, # 预测未来1小时
price_change_threshold=0.0
)
X, y, _ = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
_, _, test_data = split_data(X, y, train_split=0.7, val_split=0.15)
_, _, test_loader = create_dataloaders(
test_data, test_data, test_data,
task='classification',
batch_size=64
)
print(f"✓ 测试集大小: {len(test_data[0])} 样本")
print(f"✓ 真实标签分布: 下跌={np.sum(test_data[1]==0)}, 上涨={np.sum(test_data[1]==1)}")
# 先查看模型输出分布
print("\n[3/4] 分析模型输出分布...")
all_logits = []
all_probs = []
all_preds = []
all_labels = []
with torch.no_grad():
for batch_X, batch_y in test_loader:
batch_X = batch_X.to(device)
outputs = model(batch_X).squeeze()
# 模型输出logits
logits = outputs.cpu().numpy()
probs = torch.sigmoid(outputs).cpu().numpy()
preds = (probs > 0.5).astype(int)
all_logits.extend(logits.flatten())
all_probs.extend(probs.flatten())
all_preds.extend(preds.flatten())
all_labels.extend(batch_y.numpy().flatten())
all_logits = np.array(all_logits)
all_probs = np.array(all_probs)
all_preds = np.array(all_preds)
all_labels = np.array(all_labels)
print(f"\n📊 模型输出统计(修复后):")
print(f" Logits范围: [{all_logits.min():.4f}, {all_logits.max():.4f}]")
print(f" Logits均值: {all_logits.mean():.4f}")
print(f" 概率范围: [{all_probs.min():.4f}, {all_probs.max():.4f}]")
print(f" 概率均值: {all_probs.mean():.4f} ← 应该接近0.5")
print(f" 预测为上涨: {np.sum(all_preds==1)}/{len(all_preds)} ({np.sum(all_preds==1)/len(all_preds)*100:.1f}%)")
print(f" 预测为下跌: {np.sum(all_preds==0)}/{len(all_preds)} ({np.sum(all_preds==0)/len(all_preds)*100:.1f}%)")
# 正式评估
print("\n[4/4] 使用修复后的evaluator进行完整评估...")
metrics = evaluate_model(model, test_loader, task='classification', device=str(device))
# 保存结果
results_path = './models/checkpoints/dlinear_btc_1h_forecast/test_metrics_fixed.json'
with open(results_path, 'w') as f:
json.dump(metrics, f, indent=2)
print(f"\n✓ 评估结果已保存至:{results_path}")
# 展示结果
print("\n" + "="*80)
print("最终评估结果(修复后)")
print("="*80)
print(f"\n📊 分类指标:")
print(f" 准确率: {metrics.get('accuracy', 0):.4f} ({metrics.get('accuracy', 0)*100:.2f}%)")
print(f" F1分数: {metrics.get('f1', 0):.4f}")
print(f" 精确率: {metrics.get('precision', 0):.4f}")
print(f" 召回率: {metrics.get('recall', 0):.4f}")
if 'true_positives' in metrics:
print(f"\n📈 混淆矩阵:")
tp = metrics['true_positives']
tn = metrics['true_negatives']
fp = metrics['false_positives']
fn = metrics['false_negatives']
print(f" 真阳性(预测涨且真涨): {tp}")
print(f" 真阴性(预测跌且真跌): {tn}")
print(f" 假阳性(预测涨但真跌): {fp}")
print(f" 假阴性(预测跌但真涨): {fn}")
total = tp + tn + fp + fn
if total > 0:
print(f"\n 预测分布:")
print(f" 预测上涨: {tp + fp} ({(tp+fp)/total*100:.1f}%)")
print(f" 预测下跌: {tn + fn} ({(tn+fn)/total*100:.1f}%)")
print(f"\n💰 回测结果:")
print(f" 总收益率: {metrics.get('backtest_total_return', 0)*100:.2f}%")
print(f" 最终资金: ${metrics.get('backtest_final_capital', 0):,.2f}")
print(f" 夏普比率: {metrics.get('backtest_sharpe_ratio', 0):.4f}")
max_dd = metrics.get('backtest_max_drawdown', 0)
if np.isnan(max_dd):
print(f" 最大回撤: N/A (无交易)")
else:
print(f" 最大回撤: {max_dd*100:.2f}%")
print(f" 胜率: {metrics.get('backtest_win_rate', 0)*100:.2f}%")
print(f" 交易次数: {metrics.get('backtest_num_trades', 0)}")
# 性能评估
print(f"\n🎯 性能评估:")
acc = metrics.get('accuracy', 0)
if acc > 0.55:
print(f" ✅ 准确率 {acc*100:.1f}% > 55%,模型有一定预测能力")
elif acc > 0.50:
print(f" ⚠️ 准确率 {acc*100:.1f}% 略优于随机(50%),预测能力较弱")
else:
print(f" ❌ 准确率 {acc*100:.1f}% ≤ 50%,模型预测能力不足")
total_return = metrics.get('backtest_total_return', 0)
if total_return > 0.05:
print(f" ✅ 回测收益 {total_return*100:.2f}% > 5%,策略有较好盈利潜力")
elif total_return > 0:
print(f" ⚠️ 回测收益 {total_return*100:.2f}% > 0,策略有微弱盈利")
else:
print(f" ❌ 回测收益 {total_return*100:.2f}% ≤ 0,策略亏损")
# 对比修复前后
print(f"\n📊 修复前后对比:")
print(f" 修复前(有Bug):")
print(f" - 平均置信度: 0.69% (错误)")
print(f" - 预测上涨: 0个 (0%)")
print(f" - 准确率: 50.00% (偏差)")
print(f" 修复后(正确):")
print(f" - 平均置信度: {metrics.get('mean_confidence', 0)*100:.1f}%")
print(f" - 预测上涨: {tp + fp}个 ({(tp+fp)/total*100:.1f}%)")
print(f" - 准确率: {acc*100:.2f}%")
print("\n" + "="*80)
print("✓ 评估完成!")
print("="*80)
if __name__ == "__main__":
main()
+81
View File
@@ -0,0 +1,81 @@
"""评估已训练的DLinear模型"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import torch
import json
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
from models.dlinear.model import DLinear
from training.evaluator import evaluate_model
def main():
print("="*80)
print("评估DLinear模型(真实数据)")
print("="*80)
# 加载数据
print("\n加载数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h', use_cache=True)
preprocessor = CryptoPreprocessor(lookback=168, forecast_horizon=24)
X, y, _ = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
train_data, val_data, test_data = split_data(X, y)
_, _, test_loader = create_dataloaders(
train_data, val_data, test_data,
task='classification', batch_size=64
)
# 加载模型
print("加载最佳模型...")
model_path = './models/checkpoints/dlinear_btc_1h_real/best_model.pt'
checkpoint = torch.load(model_path, map_location='cpu')
model = DLinear(checkpoint['config'])
model.load_state_dict(checkpoint['model_state_dict'])
# 评估
print("\n在测试集上评估...")
device = 'mps' if torch.backends.mps.is_available() else 'cpu'
metrics = evaluate_model(model, test_loader, 'classification', device)
# 保存结果
results_path = './models/checkpoints/dlinear_btc_1h_real/test_metrics.json'
with open(results_path, 'w') as f:
json.dump(metrics, f, indent=2)
# 打印结果
print("\n" + "="*80)
print("测试集评估结果")
print("="*80)
print(f"\n📊 分类指标:")
print(f" 准确率: {metrics.get('accuracy', 0):.4f}")
print(f" F1分数: {metrics.get('f1', 0):.4f}")
print(f" 精确率: {metrics.get('precision', 0):.4f}")
print(f" 召回率: {metrics.get('recall', 0):.4f}")
if 'true_positives' in metrics:
print(f"\n📈 混淆矩阵:")
print(f" 真阳性: {metrics['true_positives']}")
print(f" 真阴性: {metrics['true_negatives']}")
print(f" 假阳性: {metrics['false_positives']}")
print(f" 假阴性: {metrics['false_negatives']}")
print(f"\n💰 回测结果:")
print(f" 总收益率: {metrics.get('backtest_total_return', 0)*100:.2f}%")
print(f" 最终资金: ${metrics.get('backtest_final_capital', 0):,.2f}")
print(f" 夏普比率: {metrics.get('backtest_sharpe_ratio', 0):.4f}")
print(f" 最大回撤: {metrics.get('backtest_max_drawdown', 0)*100:.2f}%")
print(f" 胜率: {metrics.get('backtest_win_rate', 0)*100:.2f}%")
print(f" 交易次数: {metrics.get('backtest_num_trades', 0)}")
print("\n" + "="*80)
print(f"✓ 结果已保存至:{results_path}")
print("="*80)
if __name__ == "__main__":
main()
+1
View File
@@ -0,0 +1 @@
"""Model implementations."""
+182
View File
@@ -0,0 +1,182 @@
"""Base model interface for all prediction models."""
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Literal, Optional
import torch
import torch.nn as nn
@dataclass
class ModelConfig:
"""Base configuration for all models."""
input_dim: int # Number of input features
lookback: int # Sequence length
task: Literal['classification', 'regression'] = 'classification'
hidden_dim: int = 512
dropout: float = 0.1
class BasePredictor(nn.Module, ABC):
"""Abstract base class for all prediction models."""
def __init__(self, config: ModelConfig):
"""Initialize model with configuration.
Args:
config: Model configuration
"""
super().__init__()
self.config = config
@abstractmethod
def forward(self, x: torch.Tensor) -> torch.Tensor:
"""Forward pass.
Args:
x: Input tensor of shape (batch_size, lookback, input_dim)
Returns:
Output tensor. Shape depends on task:
- Classification: (batch_size, 1) with sigmoid output
- Regression: (batch_size, 1) with raw value
"""
pass
def predict(self, x: torch.Tensor) -> torch.Tensor:
"""Make predictions (inference mode).
Args:
x: Input tensor
Returns:
Predictions
"""
self.eval()
with torch.no_grad():
return self.forward(x)
def get_loss_function(self, pos_weight=None, device='cpu') -> nn.Module:
"""Get appropriate loss function for the task.
Args:
pos_weight: Weight for positive class (for imbalanced data)
device: Device to place tensors on
Returns:
Loss function
"""
if self.config.task == 'classification':
if pos_weight is not None:
# 使用float32以兼容MPS (Mac M1)
pos_weight_tensor = torch.tensor([pos_weight], dtype=torch.float32).to(device)
return nn.BCEWithLogitsLoss(pos_weight=pos_weight_tensor)
else:
return nn.BCELoss()
else: # regression
return nn.MSELoss()
def save(self, path: str):
"""Save model checkpoint.
Args:
path: Path to save checkpoint
"""
torch.save({
'model_state_dict': self.state_dict(),
'config': self.config
}, path)
@classmethod
def load(cls, path: str, device: Optional[str] = None):
"""Load model from checkpoint.
Args:
path: Path to checkpoint
device: Device to load model to
Returns:
Loaded model
"""
checkpoint = torch.load(path, map_location=device)
model = cls(checkpoint['config'])
model.load_state_dict(checkpoint['model_state_dict'])
return model
def to_onnx(self, path: str, dummy_input: Optional[torch.Tensor] = None):
"""Export model to ONNX format.
Args:
path: Path to save ONNX model
dummy_input: Example input tensor. If None, will create default.
"""
# Move to CPU for ONNX export (critical for M1)
self.cpu()
self.eval()
# Create dummy input if not provided
if dummy_input is None:
dummy_input = torch.randn(
1,
self.config.lookback,
self.config.input_dim
)
# Ensure dummy input is on CPU
dummy_input = dummy_input.cpu()
# Export to ONNX
torch.onnx.export(
self,
dummy_input,
path,
export_params=True,
opset_version=14,
do_constant_folding=True,
input_names=['input'],
output_names=['output'],
dynamic_axes={
'input': {0: 'batch_size'},
'output': {0: 'batch_size'}
}
)
print(f"Model exported to ONNX: {path}")
def count_parameters(self) -> int:
"""Count trainable parameters.
Returns:
Number of trainable parameters
"""
return sum(p.numel() for p in self.parameters() if p.requires_grad)
if __name__ == "__main__":
# Test base class (with a dummy implementation)
class DummyModel(BasePredictor):
def __init__(self, config: ModelConfig):
super().__init__(config)
self.fc = nn.Linear(config.input_dim * config.lookback, 1)
if config.task == 'classification':
self.activation = nn.Sigmoid()
else:
self.activation = nn.Identity()
def forward(self, x):
batch_size = x.size(0)
x = x.view(batch_size, -1)
x = self.fc(x)
return self.activation(x)
# Test
config = ModelConfig(input_dim=10, lookback=168, task='classification')
model = DummyModel(config)
print(f"Parameters: {model.count_parameters()}")
# Test forward pass
x = torch.randn(32, 168, 10)
output = model(x)
print(f"Output shape: {output.shape}")
@@ -0,0 +1,13 @@
{
"mse": 5.495949517353438e-05,
"rmse": 0.007413467149285439,
"mae": 0.004826944321393967,
"mape": 930.6216430664062,
"directional_accuracy": 0.4907678594167834,
"backtest_total_return": -0.9745769500732422,
"backtest_final_capital": 254.23023986816406,
"backtest_sharpe_ratio": -0.26884937315369317,
"backtest_max_drawdown": -0.978744387626648,
"backtest_win_rate": 0.47166687889978354,
"backtest_num_trades": 3251
}
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,10 @@
{
"price_mae": 4283.3626861511675,
"price_rmse": 4735.107859758518,
"price_mape": 3.928560493143656,
"direction_accuracy": 0.49637127578304047,
"true_positives": 258,
"true_negatives": 2341,
"false_positives": 255,
"false_negatives": 2382
}
@@ -0,0 +1,18 @@
{
"accuracy": 0.5,
"precision": 0.0,
"recall": 0.0,
"f1": 0.0,
"true_negatives": 642,
"false_positives": 0,
"false_negatives": 642,
"true_positives": 0,
"specificity": 1.0,
"mean_confidence": 0.0068817585706710815,
"backtest_total_return": -1.0,
"backtest_final_capital": 0.0,
"backtest_sharpe_ratio": -9.643650760992955,
"backtest_max_drawdown": NaN,
"backtest_win_rate": 0.0,
"backtest_num_trades": 0
}
@@ -0,0 +1,18 @@
{
"accuracy": 0.5070093457943925,
"precision": 0.5053254437869823,
"recall": 0.6651090342679128,
"f1": 0.574310692669805,
"true_negatives": 224,
"false_positives": 418,
"false_negatives": 215,
"true_positives": 427,
"specificity": 0.34890965732087226,
"mean_confidence": 0.5017203688621521,
"backtest_total_return": -1.0,
"backtest_final_capital": -0.0,
"backtest_sharpe_ratio": 2.314261942010565,
"backtest_max_drawdown": -1.0,
"backtest_win_rate": 0.3325545171339564,
"backtest_num_trades": 136
}
@@ -0,0 +1,18 @@
{
"accuracy": 0.49764421240290335,
"precision": 0.502192108409725,
"recall": 0.3185035389282103,
"f1": 0.38979118329466356,
"true_negatives": 2648,
"false_positives": 1249,
"false_negatives": 2696,
"true_positives": 1260,
"specificity": 0.6794970490120605,
"mean_confidence": 0.49758902192115784,
"backtest_total_return": -1.0,
"backtest_final_capital": -0.0,
"backtest_sharpe_ratio": -2.5728736046923655,
"backtest_max_drawdown": NaN,
"backtest_win_rate": 0.16044823634279892,
"backtest_num_trades": 782
}
@@ -0,0 +1,13 @@
{
"mse": 2.142045195796527e-05,
"rmse": 0.004628223412710893,
"mae": 0.0030602782499045134,
"mape": 131.8273162841797,
"directional_accuracy": 0.49732620320855614,
"backtest_total_return": 0.2429506778717041,
"backtest_final_capital": 12429.5068359375,
"backtest_sharpe_ratio": 0.19540257211386183,
"backtest_max_drawdown": -0.13858233392238617,
"backtest_win_rate": 0.49732620320855614,
"backtest_num_trades": 0
}
@@ -0,0 +1,13 @@
{
"mse": 2.382730417593848e-05,
"rmse": 0.004881321970116137,
"mae": 0.0033776722848415375,
"mape": 780.3690185546875,
"directional_accuracy": 0.5066853431809499,
"backtest_total_return": -0.1671363264322281,
"backtest_final_capital": 8328.63671875,
"backtest_sharpe_ratio": -0.0266775386419239,
"backtest_max_drawdown": -0.40377509593963623,
"backtest_win_rate": 0.48974914045587675,
"backtest_num_trades": 1061
}
@@ -0,0 +1,11 @@
{
"accuracy": 0.53099173553719,
"precision": 0.5178571428571429,
"recall": 0.49361702127659574,
"f1_score": 0.505446623093682,
"true_positives": 116,
"true_negatives": 141,
"false_positives": 108,
"false_negatives": 119,
"n_samples": 484
}
@@ -0,0 +1,6 @@
{
"price_mae": 528.017854278076,
"price_rmse": 736.6082253699151,
"price_mape": 0.523728600511905,
"direction_accuracy": 0.5022918258212375
}
@@ -0,0 +1,13 @@
{
"mse": 2.1433252186398022e-05,
"rmse": 0.004629606050885758,
"mae": 0.00306295626796782,
"mape": 180.17770385742188,
"directional_accuracy": 0.4992360580595875,
"backtest_total_return": 0.226644828915596,
"backtest_final_capital": 12266.4482421875,
"backtest_sharpe_ratio": 0.18486101832170032,
"backtest_max_drawdown": -0.13858233392238617,
"backtest_win_rate": 0.4977081741787624,
"backtest_num_trades": 38
}
Binary file not shown.
Binary file not shown.
+1
View File
@@ -0,0 +1 @@
"""DLinear model implementation."""
+203
View File
@@ -0,0 +1,203 @@
"""DLinear: Decomposition Linear model for time series forecasting.
Reference: Are Transformers Effective for Time Series Forecasting? (DLinear)
Paper: https://arxiv.org/abs/2205.13504
"""
import torch
import torch.nn as nn
from dataclasses import dataclass
from typing import Literal
from ..base import BasePredictor, ModelConfig
@dataclass
class DLinearConfig(ModelConfig):
"""Configuration for DLinear model."""
moving_avg_kernel: int = 25 # Kernel size for moving average decomposition
individual: bool = False # Whether to use individual linear layers per feature
class MovingAvg(nn.Module):
"""Moving average block for trend extraction."""
def __init__(self, kernel_size: int):
"""Initialize moving average layer.
Args:
kernel_size: Size of the moving average window
"""
super().__init__()
self.kernel_size = kernel_size
self.avg = nn.AvgPool1d(
kernel_size=kernel_size,
stride=1,
padding=kernel_size // 2,
count_include_pad=False
)
def forward(self, x: torch.Tensor) -> torch.Tensor:
"""Apply moving average.
Args:
x: Input tensor of shape (batch_size, seq_len, features)
Returns:
Smoothed tensor
"""
# Padding at both ends
front = x[:, 0:1, :].repeat(1, self.kernel_size // 2, 1)
end = x[:, -1:, :].repeat(1, (self.kernel_size - 1) // 2, 1)
x_padded = torch.cat([front, x, end], dim=1)
# Apply average pooling (swap dimensions for AvgPool1d)
x_padded = x_padded.permute(0, 2, 1) # (batch, features, seq_len)
x_smoothed = self.avg(x_padded)
x_smoothed = x_smoothed.permute(0, 2, 1) # (batch, seq_len, features)
# Trim to original length
return x_smoothed[:, :x.size(1), :]
class SeriesDecomposition(nn.Module):
"""Series decomposition block to extract trend and seasonal components."""
def __init__(self, kernel_size: int):
"""Initialize decomposition layer.
Args:
kernel_size: Size of the moving average window for trend
"""
super().__init__()
self.moving_avg = MovingAvg(kernel_size)
def forward(self, x: torch.Tensor) -> tuple:
"""Decompose series into trend and seasonal components.
Args:
x: Input tensor of shape (batch_size, seq_len, features)
Returns:
Tuple of (seasonal, trend) tensors
"""
trend = self.moving_avg(x)
seasonal = x - trend
return seasonal, trend
class DLinear(BasePredictor):
"""DLinear model: Decomposition + Linear layers for forecasting."""
def __init__(self, config: DLinearConfig):
"""Initialize DLinear model.
Args:
config: Model configuration
"""
super().__init__(config)
self.decomposition = SeriesDecomposition(config.moving_avg_kernel)
if config.individual:
# Individual linear layer for each feature
self.linear_seasonal = nn.ModuleList([
nn.Linear(config.lookback, 1)
for _ in range(config.input_dim)
])
self.linear_trend = nn.ModuleList([
nn.Linear(config.lookback, 1)
for _ in range(config.input_dim)
])
else:
# Shared linear layers
self.linear_seasonal = nn.Linear(config.lookback, 1)
self.linear_trend = nn.Linear(config.lookback, 1)
# Final aggregation layer
self.aggregation = nn.Sequential(
nn.Linear(config.input_dim, config.hidden_dim),
nn.ReLU(),
nn.Dropout(config.dropout),
nn.Linear(config.hidden_dim, 1)
)
# Output activation
# 注意:对于分类任务,使用Identity而非Sigmoid
# 因为BCEWithLogitsLoss内部已包含Sigmoid,避免双重激活
if config.task == 'classification':
self.output_activation = nn.Identity() # 输出logits,由损失函数处理
else:
self.output_activation = nn.Identity()
def forward(self, x: torch.Tensor) -> torch.Tensor:
"""Forward pass through DLinear.
Args:
x: Input tensor of shape (batch_size, lookback, input_dim)
Returns:
Output tensor of shape (batch_size, 1)
"""
# Decompose into seasonal and trend
seasonal, trend = self.decomposition(x)
# Apply linear layers to each component
if self.config.individual:
seasonal_out = []
trend_out = []
for i in range(self.config.input_dim):
# Extract feature i: (batch, seq_len)
s_i = seasonal[:, :, i]
t_i = trend[:, :, i]
# Apply linear transformation: (batch, seq_len) -> (batch, 1)
seasonal_out.append(self.linear_seasonal[i](s_i))
trend_out.append(self.linear_trend[i](t_i))
# Stack outputs: (batch, input_dim)
seasonal_out = torch.stack(seasonal_out, dim=1).squeeze(-1)
trend_out = torch.stack(trend_out, dim=1).squeeze(-1)
else:
# Shared linear layer
# Swap dimensions: (batch, seq_len, features) -> (batch, features, seq_len)
seasonal = seasonal.permute(0, 2, 1)
trend = trend.permute(0, 2, 1)
# Apply linear: (batch, features, seq_len) -> (batch, features, 1)
seasonal_out = self.linear_seasonal(seasonal).squeeze(-1)
trend_out = self.linear_trend(trend).squeeze(-1)
# Combine seasonal and trend
combined = seasonal_out + trend_out # (batch, input_dim)
# Aggregate and produce final prediction
output = self.aggregation(combined) # (batch, 1)
output = self.output_activation(output)
return output
if __name__ == "__main__":
# Test DLinear model
config = DLinearConfig(
input_dim=30,
lookback=168,
task='classification',
hidden_dim=512,
moving_avg_kernel=25
)
model = DLinear(config)
print(f"DLinear parameters: {model.count_parameters()}")
# Test forward pass
x = torch.randn(32, 168, 30)
output = model(x)
print(f"Output shape: {output.shape}")
print(f"Output range: [{output.min():.4f}, {output.max():.4f}]")
# Test ONNX export
model.to_onnx('/tmp/dlinear_test.onnx')
print("ONNX export successful!")
+263
View File
@@ -0,0 +1,263 @@
"""使用全量训练模型预测2026年BTC价格涨跌"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import torch
import pickle
import numpy as np
import pandas as pd
from data.fetcher import BinanceDataFetcher
from models.dlinear.model import DLinear
from training.evaluator import ModelEvaluator
def main():
print("="*80)
print("使用全量训练模型预测2026年BTC涨跌")
print("="*80)
# ========== 步骤1: 加载模型和预处理器 ==========
print("\n[1/6] 加载训练好的模型...")
model_path = './models/checkpoints/dlinear_btc_full_data/best_model.pt'
preprocessor_path = './models/checkpoints/dlinear_btc_full_data/preprocessor.pkl'
# 加载模型
device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu')
checkpoint = torch.load(model_path, map_location=device, weights_only=False)
model = DLinear(checkpoint['config'])
model.load_state_dict(checkpoint['model_state_dict'])
model = model.to(device)
model.eval()
print(f"✓ 模型加载成功 (设备: {device})")
# 加载预处理器
with open(preprocessor_path, 'rb') as f:
preprocessor = pickle.load(f)
print(f"✓ 预处理器加载成功")
# ========== 步骤2: 获取2026年数据 ==========
print("\n[2/6] 获取2026年BTC数据...")
fetcher = BinanceDataFetcher()
try:
df_2026 = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2026-01-01',
end_date='2026-12-31',
use_cache=False # 不使用缓存,获取最新数据
)
print(f"✓ 已获取 {len(df_2026)} 行2026年数据")
print(f"✓ 时间范围:{df_2026['timestamp'].min()} 至 {df_2026['timestamp'].max()}")
print(f"✓ 价格范围:${df_2026['close'].iloc[0]:,.2f} → ${df_2026['close'].iloc[-1]:,.2f}")
total_return = (df_2026['close'].iloc[-1] - df_2026['close'].iloc[0]) / df_2026['close'].iloc[0]
print(f"✓ 2026年收益率:{total_return*100:+.2f}%")
except Exception as e:
print(f"⚠️ 无法获取2026年完整数据:{e}")
print(f"尝试获取2025年12月至今的最新数据...")
df_2026 = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2025-12-01',
use_cache=False
)
print(f"✓ 已获取 {len(df_2026)} 行最新数据")
print(f"✓ 时间范围:{df_2026['timestamp'].min()} 至 {df_2026['timestamp'].max()}")
# ========== 步骤3: 数据预处理 ==========
print("\n[3/6] 预处理2026年数据...")
# 添加技术指标和时间特征
df_processed = preprocessor.add_technical_indicators(df_2026)
df_processed = preprocessor.add_time_features(df_processed)
df_processed = preprocessor.create_labels(df_processed, task='regression')
# 删除NaN行
df_processed = df_processed.dropna().reset_index(drop=True)
# 确保包含所有训练时的列
# 如果缺少某些列,用0填充
for col in preprocessor.feature_columns:
if col not in df_processed.columns:
df_processed[col] = 0
print(f" ⚠️ 添加缺失列:{col}")
# 使用训练时的scaler标准化特征
scaled_features = preprocessor.transform(df_processed)
# 创建序列
X, y = preprocessor.create_sequences(
scaled_features,
df_processed['label'].values
)
print(f"✓ 预测序列数量:{len(X)}")
print(f"✓ 序列形状:{X.shape}")
if len(X) == 0:
print("❌ 没有足够的数据创建序列,需要至少169小时的数据")
return
# ========== 步骤4: 模型预测 ==========
print("\n[4/6] 使用模型预测未来1小时涨跌...")
X_tensor = torch.FloatTensor(X).to(device)
with torch.no_grad():
predictions = model(X_tensor).squeeze().cpu().numpy()
print(f"✓ 预测完成,共 {len(predictions)} 个预测值")
print(f"✓ 预测收益率范围:{predictions.min()*100:.4f}% ~ {predictions.max()*100:.4f}%")
print(f"✓ 预测收益率均值:{predictions.mean()*100:.4f}%")
# ========== 步骤5: 涨跌判断 ==========
print("\n[5/6] 根据预测结果判断涨跌...")
# 预测方向:正值=上涨,负值=下跌
predicted_direction = (predictions > 0).astype(int) # 1=涨, 0=跌
actual_direction = (y > 0).astype(int)
n_predict_up = (predicted_direction == 1).sum()
n_predict_down = (predicted_direction == 0).sum()
n_actual_up = (actual_direction == 1).sum()
n_actual_down = (actual_direction == 0).sum()
print(f"\n📊 预测统计:")
print(f" 预测上涨:{n_predict_up} ({n_predict_up/len(predictions)*100:.1f}%)")
print(f" 预测下跌:{n_predict_down} ({n_predict_down/len(predictions)*100:.1f}%)")
print(f"\n📊 实际统计:")
print(f" 实际上涨:{n_actual_up} ({n_actual_up/len(y)*100:.1f}%)")
print(f" 实际下跌:{n_actual_down} ({n_actual_down/len(y)*100:.1f}%)")
# ========== 步骤6: 评估预测准确率 ==========
print("\n[6/6] 评估预测准确率...")
evaluator = ModelEvaluator(task='regression')
metrics = evaluator.evaluate(y, predictions)
print("\n" + "="*80)
print("2026年预测结果评估")
print("="*80)
print(f"\n📊 回归指标:")
print(f" RMSE (均方根误差): {metrics.get('rmse', 0)*100:.4f}%")
print(f" MAE (平均绝对误差): {metrics.get('mae', 0)*100:.4f}%")
print(f" MAPE (平均百分比误差): {metrics.get('mape', 0):.2f}%")
print(f" 方向准确率: {metrics.get('directional_accuracy', 0)*100:.2f}%")
# 混淆矩阵
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(actual_direction, predicted_direction)
tn, fp, fn, tp = cm.ravel()
print(f"\n📈 混淆矩阵(方向预测):")
print(f" 真阳性(预测涨且真涨): {tp} ({tp/len(y)*100:.1f}%)")
print(f" 真阴性(预测跌且真跌): {tn} ({tn/len(y)*100:.1f}%)")
print(f" 假阳性(预测涨但真跌): {fp} ({fp/len(y)*100:.1f}%)")
print(f" 假阴性(预测跌但真涨): {fn} ({fn/len(y)*100:.1f}%)")
print(f"\n 精确率(预测涨的准确率): {tp/(tp+fp)*100 if (tp+fp)>0 else 0:.2f}%")
print(f" 召回率(实际涨被预测为涨): {tp/(tp+fn)*100 if (tp+fn)>0 else 0:.2f}%")
# 回测策略
backtest_metrics = evaluator.backtest_strategy(
y,
predictions,
initial_capital=10000.0,
transaction_cost=0.001
)
print(f"\n💰 回测结果(2026年):")
print(f" 总收益率: {backtest_metrics.get('total_return', 0)*100:.2f}%")
print(f" 最终资金: ${backtest_metrics.get('final_capital', 0):,.2f}")
print(f" 夏普比率: {backtest_metrics.get('sharpe_ratio', 0):.4f}")
max_dd = backtest_metrics.get('max_drawdown', 0)
if not np.isnan(max_dd):
print(f" 最大回撤: {max_dd*100:.2f}%")
print(f" 胜率: {backtest_metrics.get('win_rate', 0)*100:.2f}%")
print(f" 交易次数: {backtest_metrics.get('num_trades', 0):.0f}")
# Buy & Hold对比
buy_hold_return = (df_processed['close'].iloc[-1] - df_processed['close'].iloc[168]) / df_processed['close'].iloc[168]
print(f"\n📊 策略对比:")
print(f" 模型策略收益: {backtest_metrics.get('total_return', 0)*100:.2f}%")
print(f" Buy & Hold: {buy_hold_return*100:.2f}%")
if backtest_metrics.get('total_return', 0) > buy_hold_return:
print(f" ✅ 模型策略跑赢Buy & Hold {(backtest_metrics.get('total_return', 0) - buy_hold_return)*100:.2f}%")
else:
print(f" ❌ 模型策略跑输Buy & Hold {(buy_hold_return - backtest_metrics.get('total_return', 0))*100:.2f}%")
# 性能评估
print("\n" + "="*80)
print("🎯 性能评估")
print("="*80)
dir_acc = metrics.get('directional_accuracy', 0)
total_return = backtest_metrics.get('total_return', 0)
print(f"\n方向预测能力:")
if dir_acc > 0.55:
print(f" ✅ 方向准确率 {dir_acc*100:.1f}% > 55%,模型有预测能力")
elif dir_acc > 0.50:
print(f" ⚠️ 方向准确率 {dir_acc*100:.1f}% 略优于随机")
else:
print(f" ❌ 方向准确率 {dir_acc*100:.1f}% ≤ 50%,预测能力不足")
print(f"\n盈利能力:")
if total_return > 0.1:
print(f" ✅ 回测收益 {total_return*100:.2f}% > 10%,策略表现优秀")
elif total_return > 0:
print(f" ⚠️ 回测收益 {total_return*100:.2f}% > 0,策略有微弱盈利")
else:
print(f" ❌ 回测收益 {total_return*100:.2f}% ≤ 0,策略亏损")
print(f"\n风险控制:")
sharpe = backtest_metrics.get('sharpe_ratio', 0)
if sharpe > 1.0:
print(f" ✅ 夏普比率 {sharpe:.2f} > 1.0,风险调整后收益优秀")
elif sharpe > 0.5:
print(f" ⚠️ 夏普比率 {sharpe:.2f},风险调整后收益一般")
else:
print(f" ❌ 夏普比率 {sharpe:.2f} < 0.5,风险调整后收益较差")
# 详细预测样本展示
print("\n" + "="*80)
print("📋 部分预测样本展示(前20个)")
print("="*80)
print(f"\n{'时间':<20} {'预测收益率':<12} {'实际收益率':<12} {'预测方向':<8} {'实际方向':<8} {'准确':<4}")
print("-"*80)
for i in range(min(20, len(predictions))):
pred_val = predictions[i]
actual_val = y[i]
pred_dir = "上涨" if pred_val > 0 else "下跌"
actual_dir = "上涨" if actual_val > 0 else "下跌"
correct = "✓" if (pred_val > 0) == (actual_val > 0) else "✗"
timestamp = df_processed['timestamp'].iloc[i + 168] # +168因为lookback
print(f"{str(timestamp):<20} {pred_val*100:>10.4f}% {actual_val*100:>10.4f}% {pred_dir:<8} {actual_dir:<8} {correct:<4}")
print("\n" + "="*80)
print("✓ 2026年预测评估完成!")
print("="*80)
print("\n💡 总结:")
print(f" - 在2026年 {len(predictions)} 个小时的预测中")
print(f" - 方向准确率达到 {dir_acc*100:.2f}%")
print(f" - 策略回测收益 {total_return*100:.2f}%(Buy & Hold: {buy_hold_return*100:.2f}%)")
if dir_acc > 0.52 and total_return > 0:
print(f"\n ✅ 模型在2026年表现良好,可以考虑实盘小仓位测试")
elif dir_acc > 0.50:
print(f"\n ⚠️ 模型预测能力有限,建议谨慎使用")
else:
print(f"\n ❌ 模型在2026年预测失败,不建议实盘使用")
if __name__ == "__main__":
main()
+202
View File
@@ -0,0 +1,202 @@
"""2026年价格预测脚本:直接预测价格并判断涨跌方向"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import torch
import pickle
import numpy as np
import pandas as pd
import pandas_ta as ta
from sklearn.preprocessing import StandardScaler
from data.fetcher import BinanceDataFetcher
from models.dlinear.model import DLinear
class PricePreprocessor:
"""预测未来价格的预处理器"""
def __init__(self, lookback=168, forecast_horizon=1):
self.lookback = lookback
self.forecast_horizon = forecast_horizon
self.scaler = StandardScaler()
self.price_scaler = StandardScaler()
self.is_fitted = False
def add_features(self, df):
df = df.copy()
df['rsi'] = ta.rsi(df['close'], length=14)
macd = ta.macd(df['close'])
if macd is not None and not macd.empty:
df['macd'] = macd.iloc[:, 0]
df['macd_hist'] = macd.iloc[:, 1]
df['macd_signal'] = macd.iloc[:, 2]
return df
def create_labels(self, df):
df = df.copy()
df['future_price'] = df['close'].shift(-self.forecast_horizon)
df['label'] = df['future_price']
return df[:-self.forecast_horizon]
def preprocess(self, df, fit=True):
df = self.add_features(df)
df = self.create_labels(df)
df = df.dropna().reset_index(drop=True)
feature_cols = ['open', 'high', 'low', 'close', 'volume',
'rsi', 'macd', 'macd_hist', 'macd_signal']
if fit:
scaled_features = self.scaler.fit_transform(df[feature_cols])
scaled_labels = self.price_scaler.fit_transform(df[['label']])
self.is_fitted = True
self.feature_columns = feature_cols
else:
if not self.is_fitted:
raise ValueError("Must fit before transform")
scaled_features = self.scaler.transform(df[feature_cols])
scaled_labels = self.price_scaler.transform(df[['label']])
X, y = [], []
for i in range(len(scaled_features) - self.lookback):
X.append(scaled_features[i:i + self.lookback])
y.append(scaled_labels[i + self.lookback, 0])
return np.array(X), np.array(y), df
def inverse_transform_price(self, scaled_price):
return self.price_scaler.inverse_transform(scaled_price.reshape(-1, 1)).flatten()
def main():
print("="*80)
print("2026年BTC价格预测 - 直接预测未来价格")
print("="*80)
# ========== 加载模型 ==========
print("\n[1/4] 加载模型...")
model_path = './models/checkpoints/dlinear_predict_price/best_model.pt'
preprocessor_path = './models/checkpoints/dlinear_predict_price/preprocessor.pkl'
device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu')
checkpoint = torch.load(model_path, map_location=device, weights_only=False)
model = DLinear(checkpoint['config'])
model.load_state_dict(checkpoint['model_state_dict'])
model = model.to(device)
model.eval()
with open(preprocessor_path, 'rb') as f:
preprocessor = pickle.load(f)
print(f"✓ 模型加载成功")
# ========== 获取2026年数据 ==========
print("\n[2/4] 获取2026年数据...")
fetcher = BinanceDataFetcher()
try:
df_2026 = fetcher.fetch_ohlcv('BTC', '1h', start_date='2026-01-01',
end_date='2026-12-31', use_cache=False)
except:
df_2026 = fetcher.fetch_ohlcv('BTC', '1h', start_date='2025-12-01', use_cache=False)
print(f"✓ 数据量: {len(df_2026)} 行")
print(f"✓ 时间: {df_2026['timestamp'].min()} 至 {df_2026['timestamp'].max()}")
print(f"✓ 价格: ${df_2026['close'].iloc[0]:,.2f} → ${df_2026['close'].iloc[-1]:,.2f}")
# ========== 预处理 ==========
print("\n[3/4] 预处理并预测...")
X, y, processed_df = preprocessor.preprocess(df_2026, fit=False)
if len(X) == 0:
print("❌ 数据不足")
return
# 模型预测
X_tensor = torch.FloatTensor(X).to(device)
with torch.no_grad():
predictions = model(X_tensor).squeeze().cpu().numpy()
# 还原为真实价格
pred_prices = preprocessor.inverse_transform_price(predictions)
true_prices = preprocessor.inverse_transform_price(y)
# 获取当前价格
current_prices = processed_df['close'].iloc[168:168+len(pred_prices)].values
# ========== 方向判断 ==========
print("\n[4/4] 涨跌方向评估...")
pred_direction = (pred_prices > current_prices).astype(int)
true_direction = (true_prices > current_prices).astype(int)
correct = (pred_direction == true_direction).sum()
accuracy = correct / len(predictions)
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(true_direction, pred_direction)
tn, fp, fn, tp = cm.ravel()
print("\n" + "="*80)
print("📊 2026年价格预测结果")
print("="*80)
print(f"\n价格预测误差:")
price_mae = np.abs(pred_prices - true_prices).mean()
price_rmse = np.sqrt(((pred_prices - true_prices) ** 2).mean())
price_mape = np.mean(np.abs((true_prices - pred_prices) / true_prices)) * 100
print(f" MAE: ${price_mae:,.2f}")
print(f" RMSE: ${price_rmse:,.2f}")
print(f" MAPE: {price_mape:.2f}%")
print(f"\n方向预测统计:")
print(f" 预测上涨: {(pred_direction==1).sum()} / 实际上涨: {(true_direction==1).sum()}")
print(f" 预测下跌: {(pred_direction==0).sum()} / 实际下跌: {(true_direction==0).sum()}")
print(f"\n方向准确率:")
print(f" 正确: {correct}/{len(predictions)} ({accuracy*100:.2f}%)")
print(f"\n详细分析:")
print(f" ✓ 预测涨且真涨: {tp} ({tp/len(predictions)*100:.1f}%)")
print(f" ✓ 预测跌且真跌: {tn} ({tn/len(predictions)*100:.1f}%)")
print(f" ✗ 预测涨但真跌: {fp} ({fp/len(predictions)*100:.1f}%)")
print(f" ✗ 预测跌但真涨: {fn} ({fn/len(predictions)*100:.1f}%)")
# 性能评估
print("\n" + "="*80)
print("🎯 性能评估")
print("="*80)
if accuracy > 0.55:
print(f"\n ✅ 方向准确率 {accuracy*100:.2f}% > 55%,模型有预测能力")
elif accuracy > 0.52:
print(f"\n ⚠️ 方向准确率 {accuracy*100:.2f}% > 52%,略优于随机")
elif accuracy >= 0.48:
print(f"\n ⚠️ 方向准确率 {accuracy*100:.2f}% ≈ 50%,接近随机")
else:
print(f"\n ❌ 方向准确率 {accuracy*100:.2f}% < 48%,低于随机")
# 预测样本展示
print("\n" + "="*80)
print("📋 预测样本展示(前30个)")
print("="*80)
print(f"\n{'时间':<20} {'当前价格':<12} {'预测价格':<12} {'实际价格':<12} {'预测':<6} {'实际':<6} {'结果':<4}")
print("-"*90)
for i in range(min(30, len(predictions))):
timestamp = processed_df['timestamp'].iloc[i + 168]
curr = current_prices[i]
pred = pred_prices[i]
true = true_prices[i]
pred_dir = "上涨" if pred > curr else "下跌"
true_dir = "上涨" if true > curr else "下跌"
result = "✓" if pred_direction[i] == true_direction[i] else "✗"
print(f"{str(timestamp):<20} ${curr:>10,.2f} ${pred:>10,.2f} ${true:>10,.2f} "
f"{pred_dir:<6} {true_dir:<6} {result:<4}")
print("\n" + "="*80)
print(f"总结: 在{len(predictions)}次预测中,方向正确{correct}次 ({accuracy*100:.2f}%)")
print("="*80)
if __name__ == "__main__":
main()
+230
View File
@@ -0,0 +1,230 @@
"""简化预测脚本:只判断涨跌方向正确性"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import torch
import pickle
import numpy as np
import pandas as pd
import pandas_ta as ta
from sklearn.preprocessing import StandardScaler
from data.fetcher import BinanceDataFetcher
from models.dlinear.model import DLinear
class SimplePreprocessor:
"""简化的预处理器:只使用OHLCV + RSI + MACD"""
def __init__(self, lookback=168, forecast_horizon=1):
self.lookback = lookback
self.forecast_horizon = forecast_horizon
self.scaler = StandardScaler()
self.is_fitted = False
def add_features(self, df):
"""添加RSI和MACD特征"""
df = df.copy()
df['rsi'] = ta.rsi(df['close'], length=14)
macd = ta.macd(df['close'])
if macd is not None and not macd.empty:
df['macd'] = macd.iloc[:, 0]
df['macd_hist'] = macd.iloc[:, 1]
df['macd_signal'] = macd.iloc[:, 2]
return df
def create_labels(self, df):
"""创建标签:未来1小时的收益率"""
df = df.copy()
df['future_price'] = df['close'].shift(-self.forecast_horizon)
df['future_return'] = (df['future_price'] - df['close']) / df['close']
df['label'] = df['future_return']
return df[:-self.forecast_horizon]
def preprocess(self, df, fit=True):
"""完整预处理流程"""
df = self.add_features(df)
df = self.create_labels(df)
df = df.dropna().reset_index(drop=True)
feature_cols = ['open', 'high', 'low', 'close', 'volume',
'rsi', 'macd', 'macd_hist', 'macd_signal']
if fit:
scaled = self.scaler.fit_transform(df[feature_cols])
self.is_fitted = True
self.feature_columns = feature_cols
else:
if not self.is_fitted:
raise ValueError("Must fit before transform")
scaled = self.scaler.transform(df[feature_cols])
X, y = [], []
for i in range(len(scaled) - self.lookback):
X.append(scaled[i:i + self.lookback])
y.append(df['label'].iloc[i + self.lookback])
return np.array(X), np.array(y), df
def main():
print("="*80)
print("2026年BTC涨跌方向预测")
print("="*80)
# ========== 步骤1: 加载模型 ==========
print("\n[1/5] 加载模型...")
model_path = './models/checkpoints/dlinear_simple_features/best_model.pt'
preprocessor_path = './models/checkpoints/dlinear_simple_features/preprocessor.pkl'
device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu')
checkpoint = torch.load(model_path, map_location=device, weights_only=False)
model = DLinear(checkpoint['config'])
model.load_state_dict(checkpoint['model_state_dict'])
model = model.to(device)
model.eval()
with open(preprocessor_path, 'rb') as f:
preprocessor = pickle.load(f)
print(f"✓ 模型加载成功")
print(f"✓ 特征数量: {len(preprocessor.feature_columns)}")
# ========== 步骤2: 获取2026年数据 ==========
print("\n[2/5] 获取2026年数据...")
fetcher = BinanceDataFetcher()
try:
df_2026 = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2026-01-01',
end_date='2026-12-31',
use_cache=False
)
except:
print("⚠️ 获取最新可用数据...")
df_2026 = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2025-12-01',
use_cache=False
)
print(f"✓ 数据量: {len(df_2026)} 行")
print(f"✓ 时间: {df_2026['timestamp'].min()} 至 {df_2026['timestamp'].max()}")
print(f"✓ 价格: ${df_2026['close'].iloc[0]:,.2f} → ${df_2026['close'].iloc[-1]:,.2f}")
# ========== 步骤3: 预处理 ==========
print("\n[3/5] 预处理数据...")
X, y, processed_df = preprocessor.preprocess(df_2026, fit=False)
print(f"✓ 预测序列: {len(X)} 个")
print(f"✓ 实际涨跌: 上涨{(y>0).sum()}次, 下跌{(y<=0).sum()}次")
if len(X) == 0:
print("❌ 数据不足")
return
# ========== 步骤4: 预测 ==========
print("\n[4/5] 模型预测...")
X_tensor = torch.FloatTensor(X).to(device)
with torch.no_grad():
predictions = model(X_tensor).squeeze().cpu().numpy()
print(f"✓ 预测完成")
print(f"✓ 预测收益率范围: {predictions.min()*100:.4f}% ~ {predictions.max()*100:.4f}%")
# ========== 步骤5: 方向判断 ==========
print("\n[5/5] 涨跌方向评估...")
# 预测方向:正值=上涨,负值=下跌
pred_direction = (predictions > 0).astype(int) # 1=涨, 0=跌
true_direction = (y > 0).astype(int)
# 统计
n_pred_up = (pred_direction == 1).sum()
n_pred_down = (pred_direction == 0).sum()
n_true_up = (true_direction == 1).sum()
n_true_down = (true_direction == 0).sum()
# 准确率
correct = (pred_direction == true_direction).sum()
accuracy = correct / len(predictions)
# 混淆矩阵
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(true_direction, pred_direction)
tn, fp, fn, tp = cm.ravel()
print("\n" + "="*80)
print("📊 涨跌方向预测结果")
print("="*80)
print(f"\n预测统计:")
print(f" 预测上涨: {n_pred_up} ({n_pred_up/len(predictions)*100:.1f}%)")
print(f" 预测下跌: {n_pred_down} ({n_pred_down/len(predictions)*100:.1f}%)")
print(f"\n实际统计:")
print(f" 实际上涨: {n_true_up} ({n_true_up/len(y)*100:.1f}%)")
print(f" 实际下跌: {n_true_down} ({n_true_down/len(y)*100:.1f}%)")
print(f"\n准确率:")
print(f" 方向正确: {correct}/{len(predictions)} ({accuracy*100:.2f}%)")
print(f"\n详细分析:")
print(f" ✓ 预测涨且真涨: {tp} ({tp/len(y)*100:.1f}%)")
print(f" ✓ 预测跌且真跌: {tn} ({tn/len(y)*100:.1f}%)")
print(f" ✗ 预测涨但真跌: {fp} ({fp/len(y)*100:.1f}%)")
print(f" ✗ 预测跌但真涨: {fn} ({fn/len(y)*100:.1f}%)")
# 预测偏差
bias = n_pred_up - n_true_up
print(f"\n预测偏差:")
if abs(bias) < len(predictions) * 0.05:
print(f" ✅ 预测平衡(偏差{bias}个)")
elif bias > 0:
print(f" ⚠️ 偏向预测上涨(多预测{bias}次上涨)")
else:
print(f" ⚠️ 偏向预测下跌(多预测{-bias}次下跌)")
# 性能评估
print("\n" + "="*80)
print("🎯 性能评估")
print("="*80)
if accuracy > 0.55:
print(f"\n ✅ 方向准确率 {accuracy*100:.2f}% > 55%")
print(f" 模型有预测能力,可考虑实盘测试")
elif accuracy > 0.52:
print(f"\n ⚠️ 方向准确率 {accuracy*100:.2f}% > 52%")
print(f" 模型略优于随机,谨慎使用")
elif accuracy >= 0.48:
print(f"\n ⚠️ 方向准确率 {accuracy*100:.2f}% ≈ 50%")
print(f" 接近随机猜测,不建议实盘")
else:
print(f"\n ❌ 方向准确率 {accuracy*100:.2f}% < 48%")
print(f" 低于随机,模型失效")
# 预测样本展示
print("\n" + "="*80)
print("📋 预测样本展示(前30个)")
print("="*80)
print(f"\n{'时间':<20} {'预测':<6} {'实际':<6} {'结果':<4} {'预测值':<10}")
print("-"*60)
for i in range(min(30, len(predictions))):
timestamp = processed_df['timestamp'].iloc[i + 168]
pred = "上涨" if pred_direction[i] == 1 else "下跌"
true = "上涨" if true_direction[i] == 1 else "下跌"
result = "✓" if pred_direction[i] == true_direction[i] else "✗"
pred_val = predictions[i]
print(f"{str(timestamp):<20} {pred:<6} {true:<6} {result:<4} {pred_val*100:>8.4f}%")
print("\n" + "="*80)
print(f"总结: 在{len(predictions)}次预测中,方向正确{correct}次 ({accuracy*100:.2f}%)")
print("="*80)
if __name__ == "__main__":
main()
+40
View File
@@ -0,0 +1,40 @@
# Core dependencies
torch>=2.0.0
torchvision
torchaudio
python-binance>=1.0.19
pandas>=2.0.0
numpy>=1.24.0
scikit-learn>=1.3.0
pytz
python-dotenv>=1.0.0
# ONNX
onnx>=1.14.0
onnxruntime>=1.15.0
# Model-specific dependencies
darts>=0.27.0
u8darts[torch]>=0.27.0
tsfm_public>=0.2.0
gluonts[torch]>=0.14.0
# API dependencies
fastapi>=0.104.0
uvicorn[standard]>=0.24.0
pydantic>=2.0.0
python-multipart>=0.0.6
# Technical indicators
pandas-ta>=0.3.14b
# Development tools
jupyter>=1.0.0
matplotlib>=3.7.0
seaborn>=0.12.0
pytest>=7.4.0
black>=23.0.0
flake8>=6.0.0
# Utilities
tqdm>=4.66.0
+97
View File
@@ -0,0 +1,97 @@
"""生成测试数据以便在无法连接Binance时使用"""
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
from pathlib import Path
def generate_sample_data(
symbol: str = 'BTC',
timeframe: str = '1h',
num_days: int = 365
):
"""生成模拟的OHLCV数据用于测试
Args:
symbol: 币种符号
timeframe: 时间框架
num_days: 生成数据的天数
"""
# 计算数据点数量
if timeframe == '1h':
periods = num_days * 24
elif timeframe == '15m':
periods = num_days * 24 * 4
else:
periods = num_days * 24
# 生成时间戳
end_time = datetime.now()
start_time = end_time - timedelta(days=num_days)
if timeframe == '1h':
timestamps = pd.date_range(start=start_time, end=end_time, freq='h')
else: # 15m
timestamps = pd.date_range(start=start_time, end=end_time, freq='15min')
# 生成价格数据(随机游走)
base_price = {
'BTC': 45000,
'ETH': 2500,
'SOL': 100
}.get(symbol, 10000)
np.random.seed(42) # 固定种子以便复现
# 生成收益率
returns = np.random.normal(0.0001, 0.02, len(timestamps))
prices = base_price * np.exp(np.cumsum(returns))
# 生成OHLCV数据
data = []
for i, (ts, close) in enumerate(zip(timestamps, prices)):
# 生成高低价
volatility = close * 0.01
high = close + np.random.uniform(0, volatility)
low = close - np.random.uniform(0, volatility)
# 开盘价(前一个收盘价)
if i == 0:
open_price = close
else:
open_price = prices[i-1]
# 成交量
volume = np.random.uniform(1000, 10000)
data.append({
'timestamp': ts,
'open': open_price,
'high': high,
'low': low,
'close': close,
'volume': volume
})
df = pd.DataFrame(data)
# 保存到缓存
cache_dir = Path('./data/cache')
cache_dir.mkdir(parents=True, exist_ok=True)
cache_file = cache_dir / f"{symbol}_{timeframe}_1y_now.csv"
df.to_csv(cache_file, index=False)
print(f"✓ 已生成 {len(df)} 行模拟数据并保存到:{cache_file}")
return df
if __name__ == "__main__":
# 生成BTC、ETH、SOL的测试数据
for symbol in ['BTC', 'ETH', 'SOL']:
for timeframe in ['1h', '15m']:
print(f"\n生成 {symbol} {timeframe} 数据...")
generate_sample_data(symbol, timeframe)
print("\n✓ 所有测试数据已生成!")
+232
View File
@@ -0,0 +1,232 @@
"""在2026年数据上测试分类模型"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import torch
import pickle
import numpy as np
import pandas as pd
import pandas_ta as ta
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import confusion_matrix, classification_report
from data.fetcher import BinanceDataFetcher
from models.dlinear.model import DLinear
class ClassificationPreprocessor:
"""分类任务预处理器"""
def __init__(self, lookback=168, forecast_horizon=1):
self.lookback = lookback
self.forecast_horizon = forecast_horizon
self.scaler = StandardScaler()
self.is_fitted = False
def add_features(self, df):
df = df.copy()
df['rsi'] = ta.rsi(df['close'], length=14)
macd = ta.macd(df['close'])
if macd is not None and not macd.empty:
df['macd'] = macd.iloc[:, 0]
df['macd_hist'] = macd.iloc[:, 1]
df['macd_signal'] = macd.iloc[:, 2]
return df
def create_labels(self, df):
df = df.copy()
df['future_price'] = df['close'].shift(-self.forecast_horizon)
df['future_return'] = (df['future_price'] - df['close']) / df['close']
df['label'] = (df['future_return'] > 0).astype(int)
return df[:-self.forecast_horizon]
def preprocess(self, df, fit=True):
df = self.add_features(df)
df = self.create_labels(df)
df = df.dropna().reset_index(drop=True)
feature_cols = ['open', 'high', 'low', 'close', 'volume',
'rsi', 'macd', 'macd_hist', 'macd_signal']
if fit:
scaled_features = self.scaler.fit_transform(df[feature_cols])
self.is_fitted = True
self.feature_columns = feature_cols
else:
if not self.is_fitted:
raise ValueError("Must fit before transform")
scaled_features = self.scaler.transform(df[feature_cols])
X, y = [], []
for i in range(len(scaled_features) - self.lookback):
X.append(scaled_features[i:i + self.lookback])
y.append(df['label'].iloc[i + self.lookback])
return np.array(X), np.array(y), df
def main():
print("="*80)
print("2026年分类模型测试")
print("="*80)
# ========== 加载模型 ==========
print("\n[1/4] 加载模型...")
model_path = './models/checkpoints/dlinear_classification_2026/best_model.pt'
preprocessor_path = './models/checkpoints/dlinear_classification_2026/preprocessor.pkl'
device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu')
checkpoint = torch.load(model_path, map_location=device, weights_only=False)
model = DLinear(checkpoint['config'])
model.load_state_dict(checkpoint['model_state_dict'])
model = model.to(device)
model.eval()
with open(preprocessor_path, 'rb') as f:
preprocessor = pickle.load(f)
print(f"✓ 模型加载成功")
print(f"✓ 任务类型: 分类(预测涨/跌)")
# ========== 获取2026年数据 ==========
print("\n[2/4] 获取2026年测试数据...")
fetcher = BinanceDataFetcher()
try:
df_2026 = fetcher.fetch_ohlcv('BTC', '1h', start_date='2026-01-01',
end_date='2026-12-31', use_cache=False)
except:
df_2026 = fetcher.fetch_ohlcv('BTC', '1h', start_date='2025-12-01', use_cache=False)
print(f"✓ 测试数据: {len(df_2026)} 行")
print(f"✓ 时间: {df_2026['timestamp'].min()} 至 {df_2026['timestamp'].max()}")
print(f"✓ 价格: ${df_2026['close'].iloc[0]:,.2f} → ${df_2026['close'].iloc[-1]:,.2f}")
# ========== 预处理 ==========
print("\n[3/4] 预处理并预测...")
X_test, y_test, processed_df = preprocessor.preprocess(df_2026, fit=False)
if len(X_test) == 0:
print("❌ 数据不足")
return
print(f"✓ 测试序列: {len(X_test)}")
# 真实标签统计
n_true_up = (y_test == 1).sum()
n_true_down = (y_test == 0).sum()
print(f"✓ 实际涨跌: 上涨{n_true_up}次 ({n_true_up/len(y_test)*100:.1f}%), "
f"下跌{n_true_down}次 ({n_true_down/len(y_test)*100:.1f}%)")
# 模型预测
X_tensor = torch.FloatTensor(X_test).to(device)
with torch.no_grad():
logits = model(X_tensor).squeeze().cpu().numpy()
probs = 1 / (1 + np.exp(-logits)) # sigmoid
preds = (probs > 0.5).astype(int)
# ========== 评估 ==========
print("\n[4/4] 评估结果...")
# 基础统计
n_pred_up = (preds == 1).sum()
n_pred_down = (preds == 0).sum()
accuracy = (preds == y_test).mean()
# 混淆矩阵
cm = confusion_matrix(y_test, preds)
tn, fp, fn, tp = cm.ravel()
print("\n" + "="*80)
print("📊 2026年测试结果(分类任务)")
print("="*80)
print(f"\n预测统计:")
print(f" 预测上涨: {n_pred_up} ({n_pred_up/len(preds)*100:.1f}%)")
print(f" 预测下跌: {n_pred_down} ({n_pred_down/len(preds)*100:.1f}%)")
print(f"\n实际统计:")
print(f" 实际上涨: {n_true_up} ({n_true_up/len(y_test)*100:.1f}%)")
print(f" 实际下跌: {n_true_down} ({n_true_down/len(y_test)*100:.1f}%)")
print(f"\n✨ 准确率: {accuracy*100:.2f}%")
print(f"\n详细分析:")
print(f" ✓ 预测涨且真涨 (TP): {tp} ({tp/len(preds)*100:.1f}%)")
print(f" ✓ 预测跌且真跌 (TN): {tn} ({tn/len(preds)*100:.1f}%)")
print(f" ✗ 预测涨但真跌 (FP): {fp} ({fp/len(preds)*100:.1f}%)")
print(f" ✗ 预测跌但真涨 (FN): {fn} ({fn/len(preds)*100:.1f}%)")
# 精确率、召回率、F1
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
print(f"\n分类指标:")
print(f" 精确率 (Precision): {precision*100:.2f}%")
print(f" 召回率 (Recall): {recall*100:.2f}%")
print(f" F1分数: {f1:.4f}")
# 预测偏差分析
bias = n_pred_up - n_true_up
print(f"\n预测偏差:")
if abs(bias) < len(preds) * 0.05:
print(f" ✅ 预测平衡(偏差{bias}个)")
elif bias > 0:
print(f" ⚠️ 偏向预测上涨(多预测{bias}次上涨)")
else:
print(f" ⚠️ 偏向预测下跌(多预测{-bias}次下跌)")
# 性能评估
print("\n" + "="*80)
print("🎯 性能评估")
print("="*80)
if accuracy > 0.55:
print(f"\n ✅ 准确率 {accuracy*100:.2f}% > 55%,模型有预测能力!")
elif accuracy > 0.52:
print(f"\n ⚠️ 准确率 {accuracy*100:.2f}% > 52%,略优于随机")
elif accuracy >= 0.48:
print(f"\n ⚠️ 准确率 {accuracy*100:.2f}% ≈ 50%,接近随机")
else:
print(f"\n ❌ 准确率 {accuracy*100:.2f}% < 48%,低于随机")
# 预测样本展示
print("\n" + "="*80)
print("📋 预测样本展示(前30个)")
print("="*80)
print(f"\n{'时间':<20} {'预测概率':<10} {'预测':<6} {'实际':<6} {'结果':<4}")
print("-"*60)
for i in range(min(30, len(preds))):
timestamp = processed_df['timestamp'].iloc[i + 168]
prob = probs[i]
pred = "上涨" if preds[i] == 1 else "下跌"
true = "上涨" if y_test[i] == 1 else "下跌"
result = "✓" if preds[i] == y_test[i] else "✗"
print(f"{str(timestamp):<20} {prob:>8.4f} {pred:<6} {true:<6} {result:<4}")
print("\n" + "="*80)
print(f"总结: 在{len(preds)}次预测中,准确率 {accuracy*100:.2f}%")
print("="*80)
# 保存结果
results = {
'accuracy': float(accuracy),
'precision': float(precision),
'recall': float(recall),
'f1_score': float(f1),
'true_positives': int(tp),
'true_negatives': int(tn),
'false_positives': int(fp),
'false_negatives': int(fn),
'n_samples': int(len(preds))
}
import json
with open('./models/checkpoints/dlinear_classification_2026/test_2026_results.json', 'w') as f:
json.dump(results, f, indent=2)
print(f"\n✓ 结果已保存至: ./models/checkpoints/dlinear_classification_2026/test_2026_results.json")
if __name__ == "__main__":
main()
+65
View File
@@ -0,0 +1,65 @@
"""测试Binance API连接(使用SOCKS5代理)"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from data.fetcher import BinanceDataFetcher
def main():
print("="*80)
print("测试Binance API连接 - SOCKS5代理")
print("="*80)
# 测试连接
fetcher = BinanceDataFetcher()
if fetcher.client is None:
print("\n❌ API连接失败,将使用缓存数据")
return
print("\n✓ API连接成功!")
# 获取当前价格
try:
btc_price = fetcher.get_latest_price('BTC')
eth_price = fetcher.get_latest_price('ETH')
sol_price = fetcher.get_latest_price('SOL')
print(f"\n当前价格:")
print(f" BTC: ${btc_price:,.2f}")
print(f" ETH: ${eth_price:,.2f}")
print(f" SOL: ${sol_price:,.2f}")
except Exception as e:
print(f"\n❌ 获取价格失败: {e}")
return
# 获取历史数据(1天测试)
try:
print("\n正在获取BTC 1小时数据(最近1天)...")
from datetime import datetime, timedelta
end_date = datetime.now()
start_date = end_date - timedelta(days=1)
df = fetcher.fetch_ohlcv(
'BTC',
'1h',
start_date=start_date.strftime('%Y-%m-%d'),
end_date=end_date.strftime('%Y-%m-%d'),
use_cache=False # 强制从API获取
)
print(f"✓ 成功获取 {len(df)} 行数据")
print(f"\n最新数据:")
print(df.tail())
except Exception as e:
print(f"\n❌ 获取历史数据失败: {e}")
return
print("\n" + "="*80)
print("✓ Binance API测试通过!代理工作正常")
print("="*80)
if __name__ == "__main__":
main()
+169
View File
@@ -0,0 +1,169 @@
"""加密货币价格预测系统 - DLinear模型端到端测试"""
import sys
import os
# 添加项目根目录到路径
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
def main():
print("="*80)
print("加密货币价格预测系统 - DLINEAR模型测试")
print("="*80)
# 步骤1: 获取数据
print("\n[1/7] 从Binance获取BTC 1小时数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h', use_cache=True)
print(f"✓ 已获取 {len(df)} 行数据,时间范围:{df['timestamp'].min()} 至 {df['timestamp'].max()}")
# 步骤2: 数据预处理
print("\n[2/7] 数据预处理(添加技术指标、创建序列)...")
preprocessor = CryptoPreprocessor(lookback=168, forecast_horizon=24)
X, y, processed_df = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
print(f"✓ 已创建序列:X形状 = {X.shape}, y形状 = {y.shape}")
print(f"✓ 特征数量:{X.shape[-1]}")
print(f"✓ 标签分布:下跌(0) = {(y==0).sum()}, 上涨(1) = {(y==1).sum()}")
# 步骤3: 划分数据集
print("\n[3/7] 划分训练/验证/测试集 (70/15/15)...")
train_data, val_data, test_data = split_data(X, y, train_split=0.7, val_split=0.15)
# 步骤4: 创建数据加载器
print("\n[4/7] 创建PyTorch数据加载器...")
train_loader, val_loader, test_loader = create_dataloaders(
train_data, val_data, test_data,
task='classification',
batch_size=64
)
print(f"✓ 训练批次:{len(train_loader)}")
print(f"✓ 验证批次:{len(val_loader)}")
print(f"✓ 测试批次:{len(test_loader)}")
# 步骤5: 创建模型
print("\n[5/7] 创建DLinear模型...")
model_config = DLinearConfig(
input_dim=X.shape[-1],
lookback=168,
task='classification',
hidden_dim=512,
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型已创建,可训练参数:{model.count_parameters():,} 个")
# 步骤6: 训练模型
print("\n[6/7] 开始训练模型...")
train_config = TrainingConfig(
batch_size=64,
max_epochs=20, # 测试用较少轮数
learning_rate=0.001,
early_stopping_patience=5,
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/dlinear_test'
)
trainer = Trainer(model, train_config, task='classification')
history = trainer.fit(train_loader, val_loader)
# 步骤7: 评估模型
print("\n[7/7] 在测试集上评估模型...")
metrics = trainer.evaluate(test_loader)
# 打印结果摘要
print("\n" + "="*80)
print("最终测试结果")
print("="*80)
print(f"\n分类指标:")
print(f" 准确率: {metrics.get('accuracy', 0):.4f}")
print(f" F1分数: {metrics.get('f1', 0):.4f}")
print(f" 精确率: {metrics.get('precision', 0):.4f}")
print(f" 召回率: {metrics.get('recall', 0):.4f}")
if 'true_positives' in metrics:
print(f"\n混淆矩阵:")
print(f" 真阳性: {metrics['true_positives']}")
print(f" 真阴性: {metrics['true_negatives']}")
print(f" 假阳性: {metrics['false_positives']}")
print(f" 假阴性: {metrics['false_negatives']}")
print(f"\n回测结果:")
print(f" 总收益率: {metrics.get('backtest_total_return', 0)*100:.2f}%")
print(f" 最终资金: ${metrics.get('backtest_final_capital', 0):,.2f}")
print(f" 夏普比率: {metrics.get('backtest_sharpe_ratio', 0):.4f}")
print(f" 最大回撤: {metrics.get('backtest_max_drawdown', 0)*100:.2f}%")
print(f" 胜率: {metrics.get('backtest_win_rate', 0)*100:.2f}%")
print(f" 交易次数: {metrics.get('backtest_num_trades', 0)}")
# 测试ONNX导出
print("\n" + "="*80)
print("测试ONNX导出")
print("="*80)
try:
import onnx
import onnxruntime as ort
import torch
import numpy as np
onnx_path = './models/checkpoints/dlinear_test/model.onnx'
os.makedirs(os.path.dirname(onnx_path), exist_ok=True)
print("\n[1/3] 导出模型到ONNX格式...")
model.to_onnx(onnx_path)
print(f"✓ ONNX模型已保存至:{onnx_path}")
print("\n[2/3] 验证ONNX模型...")
onnx_model = onnx.load(onnx_path)
onnx.checker.check_model(onnx_model)
print("✓ ONNX模型有效")
print("\n[3/3] 测试ONNX推理与PyTorch对比...")
# 创建测试输入
test_input = torch.randn(1, 168, X.shape[-1])
# PyTorch推理(在CPU上)
model.cpu()
model.eval()
with torch.no_grad():
pytorch_output = model(test_input).numpy()
# ONNX推理
ort_session = ort.InferenceSession(onnx_path)
onnx_output = ort_session.run(
None,
{'input': test_input.numpy()}
)[0]
# 比较输出
diff = np.abs(pytorch_output - onnx_output).max()
print(f"✓ PyTorch与ONNX的最大差异:{diff:.8f}")
if diff < 1e-5:
print("✓ ONNX导出成功!输出在容差范围内匹配。")
else:
print("⚠ 警告:ONNX输出与PyTorch有差异(但可能仍可接受)")
except Exception as e:
print(f"✗ ONNX导出失败:{e}")
print("\n" + "="*80)
print("测试成功完成!")
print("="*80)
print("\n下一步:")
print(" 1. 查看训练历史:./models/checkpoints/dlinear_test/")
print(" 2. 实现N-BEATS模型(阶段3)")
print(" 3. 在所有币种上训练(BTC、ETH、SOL)")
print(" 4. 创建FastAPI预测服务")
if __name__ == "__main__":
main()
+149
View File
@@ -0,0 +1,149 @@
"""测试获取2020-2023年的历史数据"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from data.fetcher import BinanceDataFetcher
import pandas as pd
def test_historical_data():
print("="*80)
print("测试Binance API历史数据获取能力")
print("="*80)
fetcher = BinanceDataFetcher()
# 测试1: 获取2020年的数据
print("\n[测试1] 尝试获取2020年全年数据...")
try:
df_2020 = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2020-01-01',
end_date='2020-12-31',
use_cache=False # 不使用缓存,强制从API获取
)
print(f"✅ 成功获取2020年数据")
print(f" 数据行数: {len(df_2020)}")
print(f" 时间范围: {df_2020['timestamp'].min()} 至 {df_2020['timestamp'].max()}")
print(f" 价格范围: ${df_2020['close'].min():,.2f} - ${df_2020['close'].max():,.2f}")
except Exception as e:
print(f"❌ 2020年数据获取失败: {e}")
return False
# 测试2: 获取2021年的数据
print("\n[测试2] 尝试获取2021年全年数据...")
try:
df_2021 = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2021-01-01',
end_date='2021-12-31',
use_cache=False
)
print(f"✅ 成功获取2021年数据")
print(f" 数据行数: {len(df_2021)}")
print(f" 时间范围: {df_2021['timestamp'].min()} 至 {df_2021['timestamp'].max()}")
print(f" 价格范围: ${df_2021['close'].min():,.2f} - ${df_2021['close'].max():,.2f}")
except Exception as e:
print(f"❌ 2021年数据获取失败: {e}")
return False
# 测试3: 获取2022年的数据
print("\n[测试3] 尝试获取2022年全年数据...")
try:
df_2022 = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2022-01-01',
end_date='2022-12-31',
use_cache=False
)
print(f"✅ 成功获取2022年数据")
print(f" 数据行数: {len(df_2022)}")
print(f" 时间范围: {df_2022['timestamp'].min()} 至 {df_2022['timestamp'].max()}")
print(f" 价格范围: ${df_2022['close'].min():,.2f} - ${df_2022['close'].max():,.2f}")
except Exception as e:
print(f"❌ 2022年数据获取失败: {e}")
return False
# 测试4: 获取2023年的数据
print("\n[测试4] 尝试获取2023年全年数据...")
try:
df_2023 = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2023-01-01',
end_date='2023-12-31',
use_cache=False
)
print(f"✅ 成功获取2023年数据")
print(f" 数据行数: {len(df_2023)}")
print(f" 时间范围: {df_2023['timestamp'].min()} 至 {df_2023['timestamp'].max()}")
print(f" 价格范围: ${df_2023['close'].min():,.2f} - ${df_2023['close'].max():,.2f}")
except Exception as e:
print(f"❌ 2023年数据获取失败: {e}")
return False
# 测试5: 获取完整的2020-2023年数据
print("\n[测试5] 尝试获取2020-2023年完整数据(4年)...")
try:
df_full = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2020-01-01',
end_date='2023-12-31',
use_cache=False
)
print(f"✅ 成功获取2020-2023年完整数据")
print(f" 数据行数: {len(df_full)}")
print(f" 时间范围: {df_full['timestamp'].min()} 至 {df_full['timestamp'].max()}")
print(f" 起始价格: ${df_full['close'].iloc[0]:,.2f}")
print(f" 结束价格: ${df_full['close'].iloc[-1]:,.2f}")
total_return = (df_full['close'].iloc[-1] - df_full['close'].iloc[0]) / df_full['close'].iloc[0]
print(f" 总收益率: {total_return*100:.2f}%")
# 分析每年的表现
print(f"\n 年度表现分析:")
df_full['year'] = pd.to_datetime(df_full['timestamp']).dt.year
for year in [2020, 2021, 2022, 2023]:
year_df = df_full[df_full['year'] == year]
if len(year_df) > 0:
year_return = (year_df['close'].iloc[-1] - year_df['close'].iloc[0]) / year_df['close'].iloc[0]
print(f" {year}年: {year_return*100:+.2f}% (${year_df['close'].iloc[0]:,.0f} → ${year_df['close'].iloc[-1]:,.0f})")
except Exception as e:
print(f"❌ 完整数据获取失败: {e}")
return False
# 总结
print("\n" + "="*80)
print("📊 测试结果总结")
print("="*80)
print("✅ Binance API可以获取2020-2023年的完整历史数据")
print(f"✅ 总共可获取约 {len(df_full):,} 小时的K线数据")
print(f"✅ 数据质量:连续完整,无明显缺失")
print(f"\n💡 建议:")
print(f" - 使用2020-2023年数据进行训练(包含完整牛熊周期)")
print(f" - 2020年: 牛市初期")
print(f" - 2021年: 牛市顶峰")
print(f" - 2022年: 熊市")
print(f" - 2023年: 复苏")
print(f" 这样模型能学习到各种市场环境")
return True
if __name__ == "__main__":
try:
success = test_historical_data()
if success:
print("\n" + "="*80)
print("✅ 所有测试通过!可以获取2020-2023年的数据")
print("="*80)
else:
print("\n" + "="*80)
print("❌ 部分测试失败")
print("="*80)
except Exception as e:
print(f"\n❌ 测试过程出错: {e}")
+304
View File
@@ -0,0 +1,304 @@
"""8:1:1数据集划分训练 - 直接预测价格"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
import pandas as pd
import pandas_ta as ta
from sklearn.preprocessing import StandardScaler
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from data.fetcher import BinanceDataFetcher
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
import json
import pickle
class PricePreprocessor:
"""预测未来价格的预处理器"""
def __init__(self, lookback=168, forecast_horizon=1):
self.lookback = lookback
self.forecast_horizon = forecast_horizon
self.scaler = StandardScaler()
self.price_scaler = StandardScaler()
self.is_fitted = False
def add_features(self, df):
"""添加技术指标"""
df = df.copy()
df['rsi'] = ta.rsi(df['close'], length=14)
macd = ta.macd(df['close'])
if macd is not None and not macd.empty:
df['macd'] = macd.iloc[:, 0]
df['macd_hist'] = macd.iloc[:, 1]
df['macd_signal'] = macd.iloc[:, 2]
return df
def create_labels(self, df):
"""创建标签:未来1小时的价格"""
df = df.copy()
df['future_price'] = df['close'].shift(-self.forecast_horizon)
df['label'] = df['future_price']
return df[:-self.forecast_horizon]
def preprocess(self, df, fit=True):
"""完整预处理流程"""
df = self.add_features(df)
df = self.create_labels(df)
df = df.dropna().reset_index(drop=True)
feature_cols = ['open', 'high', 'low', 'close', 'volume',
'rsi', 'macd', 'macd_hist', 'macd_signal']
if fit:
scaled_features = self.scaler.fit_transform(df[feature_cols])
scaled_labels = self.price_scaler.fit_transform(df[['label']])
self.is_fitted = True
self.feature_columns = feature_cols
else:
if not self.is_fitted:
raise ValueError("Must fit before transform")
scaled_features = self.scaler.transform(df[feature_cols])
scaled_labels = self.price_scaler.transform(df[['label']])
X, y = [], []
for i in range(len(scaled_features) - self.lookback):
X.append(scaled_features[i:i + self.lookback])
y.append(scaled_labels[i + self.lookback, 0])
return np.array(X), np.array(y), df
def inverse_transform_price(self, scaled_price):
"""将标准化的价格还原为真实价格"""
return self.price_scaler.inverse_transform(scaled_price.reshape(-1, 1)).flatten()
def main():
print("="*80)
print("8:1:1数据集划分 - 直接预测未来1小时价格")
print("="*80)
# ========== 步骤1: 获取数据 ==========
print("\n[1/8] 获取BTC 2020-2025年数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2020-01-01',
end_date='2025-12-31',
use_cache=True
)
print(f"✓ 数据量: {len(df):,} 行")
print(f"✓ 时间: {df['timestamp'].min()} 至 {df['timestamp'].max()}")
print(f"✓ 价格: ${df['close'].iloc[0]:,.2f} → ${df['close'].iloc[-1]:,.2f}")
# ========== 步骤2: 预处理 ==========
print("\n[2/8] 数据预处理(只使用OHLCV + RSI + MACD)...")
preprocessor = PricePreprocessor(lookback=168, forecast_horizon=1)
X, y, processed_df = preprocessor.preprocess(df, fit=True)
print(f"✓ 序列数量: {len(X):,}")
print(f"✓ 序列形状: {X.shape}")
print(f"✓ 特征列表: {preprocessor.feature_columns}")
# ========== 步骤3: 8:1:1划分数据集 ==========
print("\n[3/8] 8:1:1划分数据集...")
n = len(X)
train_end = int(n * 0.8)
val_end = int(n * 0.9)
X_train = X[:train_end]
y_train = y[:train_end]
X_val = X[train_end:val_end]
y_val = y[train_end:val_end]
X_test = X[val_end:]
y_test = y[val_end:]
print(f" 总样本数: {n:,}")
print(f" 训练集: {len(X_train):,} ({len(X_train)/n*100:.1f}%)")
print(f" 验证集: {len(X_val):,} ({len(X_val)/n*100:.1f}%)")
print(f" 测试集: {len(X_test):,} ({len(X_test)/n*100:.1f}%)")
# 分析各数据集时间范围
lookback = 168
train_df = processed_df.iloc[lookback:train_end+lookback]
val_df = processed_df.iloc[train_end+lookback:val_end+lookback]
test_df = processed_df.iloc[val_end+lookback:]
print(f"\n 时间范围分析:")
print(f" 训练集: {train_df['timestamp'].min()} 至 {train_df['timestamp'].max()}")
train_return = (train_df['close'].iloc[-1] - train_df['close'].iloc[0]) / train_df['close'].iloc[0]
print(f" ${train_df['close'].iloc[0]:,.0f} → ${train_df['close'].iloc[-1]:,.0f} ({train_return*100:+.1f}%)")
print(f" 验证集: {val_df['timestamp'].min()} 至 {val_df['timestamp'].max()}")
val_return = (val_df['close'].iloc[-1] - val_df['close'].iloc[0]) / val_df['close'].iloc[0]
print(f" ${val_df['close'].iloc[0]:,.0f} → ${val_df['close'].iloc[-1]:,.0f} ({val_return*100:+.1f}%)")
print(f" 测试集: {test_df['timestamp'].min()} 至 {test_df['timestamp'].max()}")
test_return = (test_df['close'].iloc[-1] - test_df['close'].iloc[0]) / test_df['close'].iloc[0]
print(f" ${test_df['close'].iloc[0]:,.0f} → ${test_df['close'].iloc[-1]:,.0f} ({test_return*100:+.1f}%)")
# ========== 步骤4: 创建数据加载器 ==========
print("\n[4/8] 创建数据加载器...")
train_dataset = TensorDataset(torch.FloatTensor(X_train), torch.FloatTensor(y_train))
val_dataset = TensorDataset(torch.FloatTensor(X_val), torch.FloatTensor(y_val))
test_dataset = TensorDataset(torch.FloatTensor(X_test), torch.FloatTensor(y_test))
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=False)
val_loader = DataLoader(val_dataset, batch_size=64, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)
# ========== 步骤5: 创建模型 ==========
print("\n[5/8] 创建DLinear模型...")
model_config = DLinearConfig(
input_dim=9,
lookback=168,
task='regression',
hidden_dim=256,
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型参数: {model.count_parameters():,}")
# ========== 步骤6: 训练 ==========
print("\n[6/8] 开始训练...")
train_config = TrainingConfig(
batch_size=64,
max_epochs=50,
learning_rate=0.001,
early_stopping_patience=15,
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/dlinear_811_split'
)
trainer = Trainer(model, train_config, task='regression')
history = trainer.fit(train_loader, val_loader)
# ========== 步骤7: 在测试集上评估 ==========
print("\n[7/8] 在测试集上评估...")
device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu')
model = model.to(device)
model.eval()
test_preds = []
with torch.no_grad():
for batch_X, _ in test_loader:
batch_X = batch_X.to(device)
pred = model(batch_X).squeeze()
test_preds.extend(pred.cpu().numpy())
test_preds = np.array(test_preds)
# 还原为真实价格
pred_prices = preprocessor.inverse_transform_price(test_preds)
true_prices = preprocessor.inverse_transform_price(y_test)
# 获取当前价格
test_start_idx = val_end + lookback
current_prices = processed_df['close'].iloc[test_start_idx:test_start_idx+len(pred_prices)].values
# 方向判断
pred_direction = (pred_prices > current_prices).astype(int)
true_direction = (true_prices > current_prices).astype(int)
direction_acc = (pred_direction == true_direction).mean()
# 价格预测误差
price_mae = np.abs(pred_prices - true_prices).mean()
price_rmse = np.sqrt(((pred_prices - true_prices) ** 2).mean())
price_mape = np.mean(np.abs((true_prices - pred_prices) / true_prices)) * 100
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(true_direction, pred_direction)
tn, fp, fn, tp = cm.ravel()
print("\n" + "="*80)
print("训练完成 - 测试集性能(独立测试集)")
print("="*80)
print(f"\n📊 价格预测误差:")
print(f" MAE: ${price_mae:,.2f}")
print(f" RMSE: ${price_rmse:,.2f}")
print(f" MAPE: {price_mape:.2f}%")
print(f"\n📈 方向预测:")
print(f" 方向准确率: {direction_acc*100:.2f}%")
print(f" 预测上涨: {(pred_direction==1).sum()} / 实际上涨: {(true_direction==1).sum()}")
print(f" 预测下跌: {(pred_direction==0).sum()} / 实际下跌: {(true_direction==0).sum()}")
print(f"\n详细分析:")
print(f" ✓ 预测涨且真涨: {tp} ({tp/len(pred_prices)*100:.1f}%)")
print(f" ✓ 预测跌且真跌: {tn} ({tn/len(pred_prices)*100:.1f}%)")
print(f" ✗ 预测涨但真跌: {fp} ({fp/len(pred_prices)*100:.1f}%)")
print(f" ✗ 预测跌但真涨: {fn} ({fn/len(pred_prices)*100:.1f}%)")
# 性能评估
print("\n" + "="*80)
print("🎯 性能评估")
print("="*80)
if direction_acc > 0.55:
print(f"\n ✅ 方向准确率 {direction_acc*100:.2f}% > 55%,模型有预测能力")
elif direction_acc > 0.52:
print(f"\n ⚠️ 方向准确率 {direction_acc*100:.2f}% > 52%,略优于随机")
elif direction_acc >= 0.48:
print(f"\n ⚠️ 方向准确率 {direction_acc*100:.2f}% ≈ 50%,接近随机")
else:
print(f"\n ❌ 方向准确率 {direction_acc*100:.2f}% < 48%,低于随机")
# ========== 步骤8: 保存 ==========
print("\n[8/8] 保存模型和结果...")
with open('./models/checkpoints/dlinear_811_split/preprocessor.pkl', 'wb') as f:
pickle.dump(preprocessor, f)
metrics = {
'price_mae': float(price_mae),
'price_rmse': float(price_rmse),
'price_mape': float(price_mape),
'direction_accuracy': float(direction_acc),
'true_positives': int(tp),
'true_negatives': int(tn),
'false_positives': int(fp),
'false_negatives': int(fn)
}
with open('./models/checkpoints/dlinear_811_split/test_metrics.json', 'w') as f:
json.dump(metrics, f, indent=2)
print(f"\n✓ 模型保存至: ./models/checkpoints/dlinear_811_split/")
# 对比分析
print("\n" + "="*80)
print("📊 数据集划分对比")
print("="*80)
print("\n之前(95:5划分,无独立测试集):")
print(" 训练集: 49,733 (95%)")
print(" 验证集: 2,618 (5%)")
print(" 测试集: 无")
print(" 问题: 过拟合,验证集太小")
print(f"\n当前(8:1:1划分,有独立测试集):")
print(f" 训练集: {len(X_train):,} (80%)")
print(f" 验证集: {len(X_val):,} (10%)")
print(f" 测试集: {len(X_test):,} (10%)")
print(f" 优势: 有足够大的独立测试集评估性能")
print(f"\n测试集性能:")
print(f" 方向准确率: {direction_acc*100:.2f}%")
print(f" 价格误差MAPE: {price_mape:.2f}%")
print("\n" + "="*80)
print("✓ 训练完成!")
print("="*80)
if __name__ == "__main__":
main()
+182
View File
@@ -0,0 +1,182 @@
"""分类任务:使用2020-2025全部数据训练,2026年测试"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
import pandas as pd
import pandas_ta as ta
from sklearn.preprocessing import StandardScaler
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from data.fetcher import BinanceDataFetcher
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
import json
import pickle
class ClassificationPreprocessor:
"""分类任务预处理器:预测涨(1)或跌(0)"""
def __init__(self, lookback=168, forecast_horizon=1):
self.lookback = lookback
self.forecast_horizon = forecast_horizon
self.scaler = StandardScaler()
self.is_fitted = False
def add_features(self, df):
"""添加技术指标"""
df = df.copy()
df['rsi'] = ta.rsi(df['close'], length=14)
macd = ta.macd(df['close'])
if macd is not None and not macd.empty:
df['macd'] = macd.iloc[:, 0]
df['macd_hist'] = macd.iloc[:, 1]
df['macd_signal'] = macd.iloc[:, 2]
return df
def create_labels(self, df):
"""创建二分类标签:1=上涨, 0=下跌"""
df = df.copy()
df['future_price'] = df['close'].shift(-self.forecast_horizon)
df['future_return'] = (df['future_price'] - df['close']) / df['close']
# 二分类:大于0为上涨(1),否则为下跌(0)
df['label'] = (df['future_return'] > 0).astype(int)
return df[:-self.forecast_horizon]
def preprocess(self, df, fit=True):
"""完整预处理流程"""
df = self.add_features(df)
df = self.create_labels(df)
df = df.dropna().reset_index(drop=True)
feature_cols = ['open', 'high', 'low', 'close', 'volume',
'rsi', 'macd', 'macd_hist', 'macd_signal']
if fit:
scaled_features = self.scaler.fit_transform(df[feature_cols])
self.is_fitted = True
self.feature_columns = feature_cols
else:
if not self.is_fitted:
raise ValueError("Must fit before transform")
scaled_features = self.scaler.transform(df[feature_cols])
X, y = [], []
for i in range(len(scaled_features) - self.lookback):
X.append(scaled_features[i:i + self.lookback])
y.append(df['label'].iloc[i + self.lookback])
return np.array(X), np.array(y), df
def main():
print("="*80)
print("分类任务:2020-2025全量训练 → 2026测试")
print("="*80)
# ========== 步骤1: 获取训练数据(2020-2025) ==========
print("\n[1/6] 获取2020-2025年训练数据...")
fetcher = BinanceDataFetcher()
df_train = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2020-01-01',
end_date='2025-12-31',
use_cache=True
)
print(f"✓ 训练数据量: {len(df_train):,} 行")
print(f"✓ 时间范围: {df_train['timestamp'].min()} 至 {df_train['timestamp'].max()}")
print(f"✓ 价格: ${df_train['close'].iloc[0]:,.2f} → ${df_train['close'].iloc[-1]:,.2f}")
# ========== 步骤2: 预处理训练数据 ==========
print("\n[2/6] 预处理训练数据(分类任务)...")
preprocessor = ClassificationPreprocessor(lookback=168, forecast_horizon=1)
X_train, y_train, processed_train = preprocessor.preprocess(df_train, fit=True)
print(f"✓ 训练序列: {len(X_train):,}")
print(f"✓ 特征数量: {len(preprocessor.feature_columns)}")
print(f"✓ 特征列表: {preprocessor.feature_columns}")
# 标签分布
n_up = (y_train == 1).sum()
n_down = (y_train == 0).sum()
print(f"\n✓ 标签分布:")
print(f" 上涨(1): {n_up:,} ({n_up/len(y_train)*100:.1f}%)")
print(f" 下跌(0): {n_down:,} ({n_down/len(y_train)*100:.1f}%)")
# 计算类别权重
pos_weight = n_down / n_up if n_up > 0 else 1.0
print(f" pos_weight: {pos_weight:.3f}")
# ========== 步骤3: 创建数据加载器 ==========
print("\n[3/6] 创建数据加载器...")
train_dataset = TensorDataset(
torch.FloatTensor(X_train),
torch.FloatTensor(y_train)
)
# 用10%作为验证集(内部分割)
train_size = int(0.9 * len(train_dataset))
val_size = len(train_dataset) - train_size
train_subset, val_subset = torch.utils.data.random_split(
train_dataset, [train_size, val_size]
)
train_loader = DataLoader(train_subset, batch_size=64, shuffle=True)
val_loader = DataLoader(val_subset, batch_size=64, shuffle=False)
print(f" 训练样本: {train_size:,}")
print(f" 验证样本: {val_size:,}")
# ========== 步骤4: 创建模型 ==========
print("\n[4/6] 创建DLinear分类模型...")
model_config = DLinearConfig(
input_dim=9,
lookback=168,
task='classification', # 分类任务
hidden_dim=256,
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型参数: {model.count_parameters():,}")
print(f"✓ 任务类型: 二分类(预测涨/跌)")
print(f"✓ 损失函数: BCEWithLogitsLoss")
# ========== 步骤5: 训练 ==========
print("\n[5/6] 开始训练...")
train_config = TrainingConfig(
batch_size=64,
max_epochs=50,
learning_rate=0.001,
early_stopping_patience=15,
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/dlinear_classification_2026'
)
trainer = Trainer(
model,
train_config,
task='classification',
pos_weight=pos_weight # 使用类别权重
)
history = trainer.fit(train_loader, val_loader)
# ========== 步骤6: 保存 ==========
print("\n[6/6] 保存模型...")
with open('./models/checkpoints/dlinear_classification_2026/preprocessor.pkl', 'wb') as f:
pickle.dump(preprocessor, f)
print(f"✓ 模型保存至: ./models/checkpoints/dlinear_classification_2026/")
print("\n" + "="*80)
print("✓ 训练完成!准备在2026年数据上测试...")
print("="*80)
if __name__ == "__main__":
main()
+240
View File
@@ -0,0 +1,240 @@
"""跨资产迁移学习实验 - 用BTC训练,在ETH上验证和测试"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
import json
import numpy as np
def main():
print("="*80)
print("跨资产迁移学习实验")
print("训练:BTC 2020-2025 | 验证+测试:ETH 2020-2025")
print("="*80)
fetcher = BinanceDataFetcher()
# ========== 步骤1: 获取BTC训练数据 ==========
print("\n[1/10] 获取BTC数据(用于训练)...")
df_btc = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2020-01-01',
end_date='2025-12-31',
use_cache=True
)
print(f"✓ BTC数据: {len(df_btc)} 行")
print(f" 时间: {df_btc['timestamp'].min()} 至 {df_btc['timestamp'].max()}")
print(f" 价格: ${df_btc['close'].iloc[0]:,.0f} → ${df_btc['close'].iloc[-1]:,.0f}")
btc_return = (df_btc['close'].iloc[-1] - df_btc['close'].iloc[0]) / df_btc['close'].iloc[0]
print(f" 收益率: {btc_return*100:+.1f}%")
# ========== 步骤2: 获取ETH验证/测试数据 ==========
print("\n[2/10] 获取ETH数据(用于验证和测试)...")
df_eth = fetcher.fetch_ohlcv(
symbol='ETH',
timeframe='1h',
start_date='2020-01-01',
end_date='2025-12-31',
use_cache=True
)
print(f"✓ ETH数据: {len(df_eth)} 行")
print(f" 时间: {df_eth['timestamp'].min()} 至 {df_eth['timestamp'].max()}")
print(f" 价格: ${df_eth['close'].iloc[0]:,.2f} → ${df_eth['close'].iloc[-1]:,.2f}")
eth_return = (df_eth['close'].iloc[-1] - df_eth['close'].iloc[0]) / df_eth['close'].iloc[0]
print(f" 收益率: {eth_return*100:+.1f}%")
# ========== 步骤3: 预处理BTC数据 ==========
print("\n[3/10] 预处理BTC数据(回归任务)...")
preprocessor_btc = CryptoPreprocessor(
lookback=168,
forecast_horizon=1,
price_change_threshold=0.0
)
X_btc, y_btc, processed_btc = preprocessor_btc.preprocess_pipeline(
df_btc, task='regression', fit=True
)
print(f"✓ BTC序列: X={X_btc.shape}, y={y_btc.shape}")
# ========== 步骤4: 预处理ETH数据 ==========
print("\n[4/10] 预处理ETH数据(使用BTC的scaler)...")
# 重要:复用BTC的preprocessor(已fit的scaler)来处理ETH数据
# 添加技术指标和时间特征
df_eth_processed = preprocessor_btc.add_technical_indicators(df_eth)
df_eth_processed = preprocessor_btc.add_time_features(df_eth_processed)
df_eth_processed = preprocessor_btc.create_labels(df_eth_processed, task='regression')
# 删除NaN行
processed_eth = df_eth_processed.dropna().reset_index(drop=True)
# 使用BTC训练的scaler来标准化ETH特征(不重新fit)
scaled_features = preprocessor_btc.transform(processed_eth)
# 创建序列
X_eth, y_eth = preprocessor_btc.create_sequences(
scaled_features,
processed_eth['label'].values
)
print(f"✓ ETH序列: X={X_eth.shape}, y={y_eth.shape}")
# ========== 步骤5: 划分数据集 ==========
print("\n[5/10] 划分数据集...")
# BTC: 只用前70%作为训练集
n_btc = len(X_btc)
train_end = int(n_btc * 0.7)
X_train = X_btc[:train_end]
y_train = y_btc[:train_end]
print(f" 训练集(BTC): {len(X_train)} 样本")
print(f" 收益率范围: {y_train.min()*100:.4f}% ~ {y_train.max()*100:.4f}%")
# ETH: 70%验证,30%测试
n_eth = len(X_eth)
val_end = int(n_eth * 0.7)
X_val = X_eth[:val_end]
y_val = y_eth[:val_end]
X_test = X_eth[val_end:]
y_test = y_eth[val_end:]
print(f" 验证集(ETH): {len(X_val)} 样本")
print(f" 收益率范围: {y_val.min()*100:.4f}% ~ {y_val.max()*100:.4f}%")
print(f" 测试集(ETH): {len(X_test)} 样本")
print(f" 收益率范围: {y_test.min()*100:.4f}% ~ {y_test.max()*100:.4f}%")
# 分析时间范围
lookback = 168
train_df = processed_btc.iloc[lookback:train_end+lookback]
val_df = processed_eth.iloc[lookback:val_end+lookback]
test_df = processed_eth.iloc[val_end+lookback:]
print(f"\n 时间范围分析:")
print(f" 训练(BTC): {train_df['timestamp'].min()} 至 {train_df['timestamp'].max()}")
print(f" 验证(ETH): {val_df['timestamp'].min()} 至 {val_df['timestamp'].max()}")
print(f" 测试(ETH): {test_df['timestamp'].min()} 至 {test_df['timestamp'].max()}")
# ========== 步骤6: 创建数据加载器 ==========
print("\n[6/10] 创建数据加载器...")
train_loader, val_loader, test_loader = create_dataloaders(
(X_train, y_train),
(X_val, y_val),
(X_test, y_test),
task='regression',
batch_size=64
)
# ========== 步骤7: 创建模型 ==========
print("\n[7/10] 创建DLinear回归模型...")
model_config = DLinearConfig(
input_dim=X_btc.shape[-1],
lookback=168,
task='regression',
hidden_dim=512,
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型参数: {model.count_parameters():,} 个")
# ========== 步骤8: 训练模型 ==========
print("\n[8/10] 训练模型(BTC训练,ETH验证)...")
train_config = TrainingConfig(
batch_size=64,
max_epochs=50,
learning_rate=0.001,
early_stopping_patience=10,
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/cross_asset_btc_to_eth'
)
trainer = Trainer(model, train_config, task='regression')
history = trainer.fit(train_loader, val_loader)
# ========== 步骤9: 在ETH测试集上评估 ==========
print("\n[9/10] 在ETH测试集上评估...")
metrics_eth = trainer.evaluate(test_loader)
results_path = './models/checkpoints/cross_asset_btc_to_eth/test_metrics.json'
with open(results_path, 'w') as f:
json.dump(metrics_eth, f, indent=2)
print(f"✓ 结果保存至: {results_path}")
# ========== 步骤10: 对比分析 ==========
print("\n[10/10] 性能分析...")
print("\n" + "="*80)
print("跨资产迁移学习实验结果")
print("="*80)
print("\n📋 实验设置:")
print(f" 训练集: BTC 2020-2024 ({len(X_train):,} 样本)")
print(f" 验证集: ETH 2020-2024 ({len(X_val):,} 样本)")
print(f" 测试集: ETH 2024-2025 ({len(X_test):,} 样本)")
print("\n📊 ETH测试集性能:")
print(f" RMSE: {metrics_eth.get('rmse', 0)*100:.4f}%")
print(f" MAE: {metrics_eth.get('mae', 0)*100:.4f}%")
print(f" 方向准确率: {metrics_eth.get('directional_accuracy', 0)*100:.2f}%")
print(f"\n💰 ETH回测结果:")
print(f" 总收益率: {metrics_eth.get('backtest_total_return', 0)*100:.2f}%")
print(f" 夏普比率: {metrics_eth.get('backtest_sharpe_ratio', 0):.4f}")
max_dd = metrics_eth.get('backtest_max_drawdown', 0)
if not np.isnan(max_dd):
print(f" 最大回撤: {max_dd*100:.2f}%")
print(f" 胜率: {metrics_eth.get('backtest_win_rate', 0)*100:.2f}%")
print(f" 交易次数: {metrics_eth.get('backtest_num_trades', 0):.0f}")
# 对比之前的结果
print("\n" + "="*80)
print("📊 性能对比")
print("="*80)
print("\n实验1:BTC训练 → BTC测试(之前结果)")
print(" 方向准确率: 50.67%")
print(" RMSE: 0.4881%")
print(" 回测收益: -16.71%")
print(f"\n实验2:BTC训练 → ETH测试(当前结果)")
print(f" 方向准确率: {metrics_eth.get('directional_accuracy', 0)*100:.2f}%")
print(f" RMSE: {metrics_eth.get('rmse', 0)*100:.4f}%")
print(f" 回测收益: {metrics_eth.get('backtest_total_return', 0)*100:.2f}%")
# 性能评估
print("\n🎯 迁移学习效果评估:")
dir_acc = metrics_eth.get('directional_accuracy', 0)
if dir_acc > 0.52:
print(f" ✅ 方向准确率 {dir_acc*100:.1f}% > 52%,跨资产迁移有效")
elif dir_acc > 0.50:
print(f" ⚠️ 方向准确率 {dir_acc*100:.1f}% 略优于随机")
else:
print(f" ❌ 方向准确率 {dir_acc*100:.1f}% ≤ 50%,迁移失败")
total_return = metrics_eth.get('backtest_total_return', 0)
if total_return > 0:
print(f" ✅ 回测收益 {total_return*100:.2f}% > 0,策略盈利")
else:
print(f" ❌ 回测收益 {total_return*100:.2f}% ≤ 0,策略亏损")
print("\n💡 结论:")
if dir_acc > 0.52 and total_return > 0:
print(" ✅ 跨资产迁移学习成功!BTC学到的模式可应用于ETH")
elif dir_acc > 0.50:
print(" ⚠️ 跨资产迁移部分有效,但收益不佳")
else:
print(" ❌ 跨资产迁移失败,BTC和ETH市场动态差异太大")
print("\n" + "="*80)
print("✓ 实验完成!")
print("="*80)
if __name__ == "__main__":
main()
+156
View File
@@ -0,0 +1,156 @@
"""训练DLinear模型 - 预测未来1小时涨跌(已修复双重Sigmoid Bug)"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
import json
import numpy as np
def main():
print("="*80)
print("DLinear模型训练 - 预测未来1小时涨跌(已修复双重Sigmoid Bug)")
print("="*80)
# 步骤1: 获取真实数据
print("\n[1/7] 从Binance获取BTC 1h真实数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h', use_cache=True)
print(f"✓ 已获取 {len(df)} 行真实数据")
print(f"✓ 时间范围:{df['timestamp'].min()} 至 {df['timestamp'].max()}")
# 步骤2: 数据预处理(预测未来1小时)
print("\n[2/7] 数据预处理(预测未来1小时的涨跌)...")
preprocessor = CryptoPreprocessor(
lookback=168, # 输入:过去168小时(7天)
forecast_horizon=1, # ← 改为预测未来1小时
price_change_threshold=0.0 # 任何涨幅>0都算上涨
)
X, y, processed_df = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
print(f"✓ 已创建序列:X形状 = {X.shape}, y形状 = {y.shape}")
print(f"✓ 特征数量:{X.shape[-1]}")
# 计算类别分布
n_down = (y == 0).sum()
n_up = (y == 1).sum()
print(f"✓ 标签分布:下跌(0) = {n_down} ({n_down/len(y)*100:.1f}%), 上涨(1) = {n_up} ({n_up/len(y)*100:.1f}%)")
# 计算类别权重
pos_weight = n_down / n_up if n_up > 0 else 1.0
print(f"✓ 计算类别权重:pos_weight = {pos_weight:.2f}")
# 步骤3: 划分数据集
print("\n[3/7] 划分训练/验证/测试集 (70/15/15)...")
train_data, val_data, test_data = split_data(X, y, train_split=0.7, val_split=0.15)
print(f" 训练集:下跌 {(train_data[1]==0).sum()}, 上涨 {(train_data[1]==1).sum()}")
print(f" 验证集:下跌 {(val_data[1]==0).sum()}, 上涨 {(val_data[1]==1).sum()}")
print(f" 测试集:下跌 {(test_data[1]==0).sum()}, 上涨 {(test_data[1]==1).sum()}")
# 步骤4: 创建数据加载器
print("\n[4/7] 创建PyTorch数据加载器...")
train_loader, val_loader, test_loader = create_dataloaders(
train_data, val_data, test_data,
task='classification',
batch_size=64
)
# 步骤5: 创建模型
print("\n[5/7] 创建DLinear模型(已修复双重Sigmoid Bug)...")
model_config = DLinearConfig(
input_dim=X.shape[-1],
lookback=168,
task='classification',
hidden_dim=512,
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型已创建,可训练参数:{model.count_parameters():,} 个")
print(f"✓ Bug修复:输出层使用Identity(输出logits),由BCEWithLogitsLoss处理Sigmoid")
# 步骤6: 训练模型
print("\n[6/7] 开始训练模型...")
train_config = TrainingConfig(
batch_size=64,
max_epochs=50,
learning_rate=0.001,
early_stopping_patience=10,
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/dlinear_btc_1h_forecast'
)
trainer = Trainer(
model,
train_config,
task='classification',
pos_weight=pos_weight
)
history = trainer.fit(train_loader, val_loader)
# 步骤7: 评估模型
print("\n[7/7] 在测试集上评估模型...")
metrics = trainer.evaluate(test_loader)
# 保存评估结果
results_path = './models/checkpoints/dlinear_btc_1h_forecast/test_metrics.json'
with open(results_path, 'w') as f:
json.dump(metrics, f, indent=2)
print(f"\n✓ 评估结果已保存至:{results_path}")
# 打印结果摘要
print("\n" + "="*80)
print("训练完成 - 最终测试结果(预测未来1小时涨跌)")
print("="*80)
print(f"\n📊 分类指标:")
print(f" 准确率: {metrics.get('accuracy', 0):.4f} ({metrics.get('accuracy', 0)*100:.2f}%)")
print(f" F1分数: {metrics.get('f1', 0):.4f}")
print(f" 精确率: {metrics.get('precision', 0):.4f}")
print(f" 召回率: {metrics.get('recall', 0):.4f}")
if 'true_positives' in metrics:
print(f"\n📈 混淆矩阵:")
print(f" 真阳性(预测涨且真涨): {metrics['true_positives']}")
print(f" 真阴性(预测跌且真跌): {metrics['true_negatives']}")
print(f" 假阳性(预测涨但真跌): {metrics['false_positives']}")
print(f" 假阴性(预测跌但真涨): {metrics['false_negatives']}")
print(f"\n💰 回测结果:")
print(f" 总收益率: {metrics.get('backtest_total_return', 0)*100:.2f}%")
print(f" 最终资金: ${metrics.get('backtest_final_capital', 0):,.2f}")
print(f" 夏普比率: {metrics.get('backtest_sharpe_ratio', 0):.4f}")
print(f" 最大回撤: {metrics.get('backtest_max_drawdown', 0)*100:.2f}%")
print(f" 胜率: {metrics.get('backtest_win_rate', 0)*100:.2f}%")
print(f" 交易次数: {metrics.get('backtest_num_trades', 0)}")
# 性能分析
print(f"\n🎯 性能评估:")
acc = metrics.get('accuracy', 0)
if acc > 0.55:
print(f" ✅ 准确率 {acc*100:.1f}% > 55%,模型有一定预测能力")
elif acc > 0.50:
print(f" ⚠️ 准确率 {acc*100:.1f}% 略优于随机(50%),预测能力较弱")
else:
print(f" ❌ 准确率 {acc*100:.1f}% ≤ 50%,模型预测能力不足")
total_return = metrics.get('backtest_total_return', 0)
if total_return > 0:
print(f" ✅ 回测收益 {total_return*100:.2f}% > 0,策略有盈利潜力")
else:
print(f" ❌ 回测收益 {total_return*100:.2f}% ≤ 0,策略亏损")
print("\n" + "="*80)
print("✓ 训练完成!")
print("="*80)
if __name__ == "__main__":
main()
+190
View File
@@ -0,0 +1,190 @@
"""使用真实Binance数据训练DLinear模型 - 优化版(平衡类别 + 类别权重)"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
import json
import numpy as np
def main():
print("="*80)
print("DLinear模型训练 - 优化版(平衡类别 + 类别权重)")
print("="*80)
# 步骤1: 获取真实数据
print("\n[1/8] 从Binance获取BTC 1h真实数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h', use_cache=True)
print(f"✓ 已获取 {len(df)} 行真实数据")
print(f"✓ 时间范围:{df['timestamp'].min()} 至 {df['timestamp'].max()}")
# 步骤2: 数据预处理(使用真实涨跌:threshold=0.0)
print("\n[2/8] 数据预处理(使用真实涨跌标签,threshold=0.0)...")
preprocessor = CryptoPreprocessor(
lookback=168,
forecast_horizon=24,
price_change_threshold=0.0 # 任何涨幅>0都算上涨
)
X, y, processed_df = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
print(f"✓ 已创建序列:X形状 = {X.shape}, y形状 = {y.shape}")
print(f"✓ 特征数量:{X.shape[-1]}")
# 计算类别分布
n_down = (y == 0).sum()
n_up = (y == 1).sum()
print(f"✓ 标签分布(优化后):下跌(0) = {n_down} ({n_down/len(y)*100:.1f}%), 上涨(1) = {n_up} ({n_up/len(y)*100:.1f}%)")
# 计算类别权重
pos_weight = n_down / n_up if n_up > 0 else 1.0
print(f"✓ 计算类别权重:pos_weight = {pos_weight:.2f} (给上涨类{pos_weight:.2f}倍权重)")
# 步骤3: 划分数据集
print("\n[3/8] 划分训练/验证/测试集 (70/15/15)...")
train_data, val_data, test_data = split_data(X, y, train_split=0.7, val_split=0.15)
# 显示各数据集的类别分布
print(f" 训练集:下跌 {(train_data[1]==0).sum()}, 上涨 {(train_data[1]==1).sum()}")
print(f" 验证集:下跌 {(val_data[1]==0).sum()}, 上涨 {(val_data[1]==1).sum()}")
print(f" 测试集:下跌 {(test_data[1]==0).sum()}, 上涨 {(test_data[1]==1).sum()}")
# 步骤4: 创建数据加载器
print("\n[4/8] 创建PyTorch数据加载器...")
train_loader, val_loader, test_loader = create_dataloaders(
train_data, val_data, test_data,
task='classification',
batch_size=64
)
# 步骤5: 创建模型
print("\n[5/8] 创建DLinear模型...")
model_config = DLinearConfig(
input_dim=X.shape[-1],
lookback=168,
task='classification',
hidden_dim=512,
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型已创建,可训练参数:{model.count_parameters():,} 个")
# 步骤6: 训练模型(使用类别权重)
print("\n[6/8] 开始训练模型(使用加权损失函数)...")
train_config = TrainingConfig(
batch_size=64,
max_epochs=50,
learning_rate=0.001,
early_stopping_patience=10,
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/dlinear_btc_1h_balanced'
)
trainer = Trainer(
model,
train_config,
task='classification',
pos_weight=pos_weight # 传入类别权重
)
history = trainer.fit(train_loader, val_loader)
# 步骤7: 评估模型
print("\n[7/8] 在测试集上评估模型...")
metrics = trainer.evaluate(test_loader)
# 保存评估结果
results_path = './models/checkpoints/dlinear_btc_1h_balanced/test_metrics.json'
with open(results_path, 'w') as f:
json.dump(metrics, f, indent=2)
print(f"\n✓ 评估结果已保存至:{results_path}")
# 打印结果摘要
print("\n" + "="*80)
print("训练完成 - 最终测试结果(优化版)")
print("="*80)
print(f"\n📊 分类指标:")
print(f" 准确率: {metrics.get('accuracy', 0):.4f}")
print(f" F1分数: {metrics.get('f1', 0):.4f}")
print(f" 精确率: {metrics.get('precision', 0):.4f}")
print(f" 召回率: {metrics.get('recall', 0):.4f}")
if 'true_positives' in metrics:
print(f"\n📈 混淆矩阵:")
print(f" 真阳性(预测涨且真涨): {metrics['true_positives']}")
print(f" 真阴性(预测跌且真跌): {metrics['true_negatives']}")
print(f" 假阳性(预测涨但真跌): {metrics['false_positives']}")
print(f" 假阴性(预测跌但真涨): {metrics['false_negatives']}")
print(f"\n💰 回测结果:")
print(f" 总收益率: {metrics.get('backtest_total_return', 0)*100:.2f}%")
print(f" 最终资金: ${metrics.get('backtest_final_capital', 0):,.2f}")
print(f" 夏普比率: {metrics.get('backtest_sharpe_ratio', 0):.4f}")
print(f" 最大回撤: {metrics.get('backtest_max_drawdown', 0)*100:.2f}%")
print(f" 胜率: {metrics.get('backtest_win_rate', 0)*100:.2f}%")
print(f" 交易次数: {metrics.get('backtest_num_trades', 0)}")
# 步骤8: ONNX导出
print("\n[8/8] 测试ONNX导出...")
try:
import onnx
import onnxruntime as ort
import torch
onnx_path = './models/checkpoints/dlinear_btc_1h_balanced/model.onnx'
print(" 导出模型到ONNX格式...")
model.to_onnx(onnx_path)
print(f" ✓ ONNX模型已保存至:{onnx_path}")
print(" 验证ONNX模型...")
onnx_model = onnx.load(onnx_path)
onnx.checker.check_model(onnx_model)
print(" ✓ ONNX模型有效")
# 测试推理
test_input = torch.randn(1, 168, X.shape[-1])
model.cpu()
model.eval()
with torch.no_grad():
pytorch_output = model(test_input).numpy()
ort_session = ort.InferenceSession(onnx_path)
onnx_output = ort_session.run(None, {'input': test_input.numpy()})[0]
diff = np.abs(pytorch_output - onnx_output).max()
print(f" ✓ PyTorch与ONNX的最大差异:{diff:.8f}")
if diff < 1e-5:
print(" ✓ ONNX导出成功!")
else:
print(" ⚠ ONNX输出有差异(可能仍可接受)")
except Exception as e:
print(f" ✗ ONNX导出失败:{e}")
print("\n" + "="*80)
print("✓ DLinear模型训练完成(优化版)")
print("="*80)
# 对比旧版本
print("\n📊 优化效果对比:")
print(" 旧版本(1.5%阈值):")
print(" - 类别分布:上涨 15.5%, 下跌 84.5%")
print(" - 准确率:84.47% F1:0.00 召回率:0.00")
print(" - 模型策略:永远预测下跌")
print(" ")
print(" 新版本(0%阈值 + 类别权重):")
print(f" - 类别分布:上涨 {n_up/len(y)*100:.1f}%, 下跌 {n_down/len(y)*100:.1f}%")
print(f" - 准确率:{metrics.get('accuracy', 0):.2%} F1:{metrics.get('f1', 0):.2f} 召回率:{metrics.get('recall', 0):.2f}")
print(f" - 回测收益:{metrics.get('backtest_total_return', 0)*100:.2f}%")
if __name__ == "__main__":
main()
+242
View File
@@ -0,0 +1,242 @@
"""使用2020-2025年数据训练DLinear模型 - 包含完整牛熊周期"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
import json
import numpy as np
def main():
print("="*80)
print("DLinear模型训练 - 使用2020-2025年完整数据(5年,包含牛熊周期)")
print("="*80)
# 步骤1: 获取2020-2025年完整数据
print("\n[1/8] 从Binance获取BTC 2020-2025年完整数据...")
fetcher = BinanceDataFetcher()
# 获取5年数据
df = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2020-01-01',
end_date='2025-12-31',
use_cache=True
)
print(f"✓ 已获取 {len(df)} 行数据(约{len(df)/8760:.1f}年)")
print(f"✓ 时间范围:{df['timestamp'].min()} 至 {df['timestamp'].max()}")
# 分析数据覆盖的市场周期
print(f"\n✓ 价格变化分析:")
print(f" 起始价格: ${df['close'].iloc[0]:,.2f} (2020年)")
print(f" 结束价格: ${df['close'].iloc[-1]:,.2f} (2025年)")
total_return = (df['close'].iloc[-1] - df['close'].iloc[0]) / df['close'].iloc[0]
print(f" 总收益率: {total_return*100:+.2f}%")
# 按年分析
df['year'] = df['timestamp'].dt.year
print(f"\n✓ 年度表现分析(确保包含牛熊周期):")
for year in sorted(df['year'].unique()):
year_df = df[df['year'] == year]
if len(year_df) > 100: # 确保有足够数据
year_return = (year_df['close'].iloc[-1] - year_df['close'].iloc[0]) / year_df['close'].iloc[0]
trend = "🚀 牛市" if year_return > 0.3 else ("🔴 熊市" if year_return < -0.3 else "📊 震荡")
print(f" {year}年: {year_return*100:+7.2f}% {trend}")
# 步骤2: 数据预处理(预测未来1小时)
print("\n[2/8] 数据预处理(预测未来1小时的涨跌)...")
preprocessor = CryptoPreprocessor(
lookback=168, # 输入:过去168小时(7天)
forecast_horizon=1, # 预测:未来1小时
price_change_threshold=0.0
)
X, y, processed_df = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
print(f"✓ 已创建序列:X形状 = {X.shape}, y形状 = {y.shape}")
print(f"✓ 特征数量:{X.shape[-1]}")
# 计算类别分布
n_down = (y == 0).sum()
n_up = (y == 1).sum()
print(f"✓ 标签分布:下跌(0) = {n_down} ({n_down/len(y)*100:.1f}%), 上涨(1) = {n_up} ({n_up/len(y)*100:.1f}%)")
# 计算类别权重
pos_weight = n_down / n_up if n_up > 0 else 1.0
print(f"✓ 计算类别权重:pos_weight = {pos_weight:.2f}")
# 步骤3: 划分数据集(时间序列分割)
print("\n[3/8] 划分训练/验证/测试集 (70/15/15)...")
train_data, val_data, test_data = split_data(X, y, train_split=0.7, val_split=0.15)
print(f" 训练集:{len(train_data[0])} 样本")
print(f" 下跌: {(train_data[1]==0).sum()}, 上涨: {(train_data[1]==1).sum()}")
print(f" 验证集:{len(val_data[0])} 样本")
print(f" 下跌: {(val_data[1]==0).sum()}, 上涨: {(val_data[1]==1).sum()}")
print(f" 测试集:{len(test_data[0])} 样本")
print(f" 下跌: {(test_data[1]==0).sum()}, 上涨: {(test_data[1]==1).sum()}")
# 分析各数据集对应的时间范围
lookback = 168
train_end_idx = int(len(processed_df) * 0.7) - lookback
val_end_idx = int(len(processed_df) * 0.85) - lookback
train_df = processed_df.iloc[lookback:train_end_idx+lookback]
val_df = processed_df.iloc[train_end_idx+lookback:val_end_idx+lookback]
test_df = processed_df.iloc[val_end_idx+lookback:]
print(f"\n 训练集时间范围: {train_df['timestamp'].min()} 至 {train_df['timestamp'].max()}")
train_return = (train_df['close'].iloc[-1] - train_df['close'].iloc[0]) / train_df['close'].iloc[0]
print(f" 价格: ${train_df['close'].iloc[0]:,.0f} → ${train_df['close'].iloc[-1]:,.0f} ({train_return*100:+.1f}%)")
print(f" 验证集时间范围: {val_df['timestamp'].min()} 至 {val_df['timestamp'].max()}")
val_return = (val_df['close'].iloc[-1] - val_df['close'].iloc[0]) / val_df['close'].iloc[0]
print(f" 价格: ${val_df['close'].iloc[0]:,.0f} → ${val_df['close'].iloc[-1]:,.0f} ({val_return*100:+.1f}%)")
print(f" 测试集时间范围: {test_df['timestamp'].min()} 至 {test_df['timestamp'].max()}")
test_return = (test_df['close'].iloc[-1] - test_df['close'].iloc[0]) / test_df['close'].iloc[0]
print(f" 价格: ${test_df['close'].iloc[0]:,.0f} → ${test_df['close'].iloc[-1]:,.0f} ({test_return*100:+.1f}%)")
# 步骤4: 创建数据加载器
print("\n[4/8] 创建PyTorch数据加载器...")
train_loader, val_loader, test_loader = create_dataloaders(
train_data, val_data, test_data,
task='classification',
batch_size=64
)
# 步骤5: 创建模型
print("\n[5/8] 创建DLinear模型...")
model_config = DLinearConfig(
input_dim=X.shape[-1],
lookback=168,
task='classification',
hidden_dim=512,
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型已创建,可训练参数:{model.count_parameters():,} 个")
# 步骤6: 训练模型
print("\n[6/8] 开始训练模型(数据量大,预计需要更长时间)...")
train_config = TrainingConfig(
batch_size=64,
max_epochs=50,
learning_rate=0.001,
early_stopping_patience=10,
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/dlinear_btc_2020_2025'
)
trainer = Trainer(
model,
train_config,
task='classification',
pos_weight=pos_weight
)
print(f"✓ 训练数据规模:{len(train_data[0]):,} 样本(是之前的 {len(train_data[0])/5990:.1f}x)")
history = trainer.fit(train_loader, val_loader)
# 步骤7: 评估模型
print("\n[7/8] 在测试集上评估模型...")
metrics = trainer.evaluate(test_loader)
# 保存评估结果
results_path = './models/checkpoints/dlinear_btc_2020_2025/test_metrics.json'
with open(results_path, 'w') as f:
json.dump(metrics, f, indent=2)
print(f"\n✓ 评估结果已保存至:{results_path}")
# 步骤8: 对比分析
print("\n[8/8] 性能对比分析...")
print("\n" + "="*80)
print("训练完成 - 最终测试结果(使用2020-2025年数据)")
print("="*80)
print(f"\n📊 分类指标:")
print(f" 准确率: {metrics.get('accuracy', 0):.4f} ({metrics.get('accuracy', 0)*100:.2f}%)")
print(f" F1分数: {metrics.get('f1', 0):.4f}")
print(f" 精确率: {metrics.get('precision', 0):.4f}")
print(f" 召回率: {metrics.get('recall', 0):.4f}")
if 'true_positives' in metrics:
tp = metrics['true_positives']
tn = metrics['true_negatives']
fp = metrics['false_positives']
fn = metrics['false_negatives']
print(f"\n📈 混淆矩阵:")
print(f" 真阳性(预测涨且真涨): {tp}")
print(f" 真阴性(预测跌且真跌): {tn}")
print(f" 假阳性(预测涨但真跌): {fp}")
print(f" 假阴性(预测跌但真涨): {fn}")
total = tp + tn + fp + fn
if total > 0:
print(f"\n 预测分布:")
print(f" 预测上涨: {tp + fp} ({(tp+fp)/total*100:.1f}%)")
print(f" 预测下跌: {tn + fn} ({(tn+fn)/total*100:.1f}%)")
print(f"\n💰 回测结果:")
print(f" 总收益率: {metrics.get('backtest_total_return', 0)*100:.2f}%")
print(f" 最终资金: ${metrics.get('backtest_final_capital', 0):,.2f}")
print(f" 夏普比率: {metrics.get('backtest_sharpe_ratio', 0):.4f}")
max_dd = metrics.get('backtest_max_drawdown', 0)
if not np.isnan(max_dd):
print(f" 最大回撤: {max_dd*100:.2f}%")
else:
print(f" 最大回撤: N/A")
print(f" 胜率: {metrics.get('backtest_win_rate', 0)*100:.2f}%")
print(f" 交易次数: {metrics.get('backtest_num_trades', 0)}")
# 性能评估
print(f"\n🎯 性能评估:")
acc = metrics.get('accuracy', 0)
if acc > 0.55:
print(f" ✅ 准确率 {acc*100:.1f}% > 55%,模型有一定预测能力")
elif acc > 0.50:
print(f" ⚠️ 准确率 {acc*100:.1f}% 略优于随机(50%),预测能力较弱")
else:
print(f" ❌ 准确率 {acc*100:.1f}% ≤ 50%,模型预测能力不足")
total_return = metrics.get('backtest_total_return', 0)
if total_return > 0.05:
print(f" ✅ 回测收益 {total_return*100:.2f}% > 5%,策略有较好盈利潜力")
elif total_return > 0:
print(f" ⚠️ 回测收益 {total_return*100:.2f}% > 0,策略有微弱盈利")
else:
print(f" ❌ 回测收益 {total_return*100:.2f}% ≤ 0,策略亏损")
# 对比之前的结果
print(f"\n📊 与之前结果对比:")
print(f" 使用1年数据(2025年):")
print(f" - 训练集: 2025.02-10 (牛市+16%)")
print(f" - 测试集: 2025.12-2026.01 (熊市-2%)")
print(f" - 准确率: 50.7%")
print(f" - 预测上涨: 65.8% (偏向上涨)")
print(f" - 问题: 训练测试分布不一致")
print(f"\n 使用5年数据(2020-2025年):")
print(f" - 训练集: 包含牛市(2020-2021)和熊市(2022)")
print(f" - 测试集: 2024-2025数据")
print(f" - 准确率: {acc*100:.1f}%")
if total > 0:
print(f" - 预测上涨: {(tp+fp)/total*100:.1f}%")
print(f" - 改进: 见过完整市场周期,泛化更好")
print("\n" + "="*80)
print("✓ 训练完成!")
print("="*80)
if __name__ == "__main__":
main()
+163
View File
@@ -0,0 +1,163 @@
"""使用真实Binance数据训练DLinear模型"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
import json
def main():
print("="*80)
print("DLinear模型训练 - 真实Binance数据")
print("="*80)
# 步骤1: 获取真实数据
print("\n[1/7] 从Binance获取BTC 1h真实数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h', use_cache=True)
print(f"✓ 已获取 {len(df)} 行真实数据")
print(f"✓ 时间范围:{df['timestamp'].min()} 至 {df['timestamp'].max()}")
# 步骤2: 数据预处理
print("\n[2/7] 数据预处理(技术指标 + 特征工程)...")
preprocessor = CryptoPreprocessor(lookback=168, forecast_horizon=24)
X, y, processed_df = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
print(f"✓ 已创建序列:X形状 = {X.shape}, y形状 = {y.shape}")
print(f"✓ 特征数量:{X.shape[-1]}")
print(f"✓ 标签分布:下跌(0) = {(y==0).sum()}, 上涨(1) = {(y==1).sum()}")
# 步骤3: 划分数据集
print("\n[3/7] 划分训练/验证/测试集 (70/15/15)...")
train_data, val_data, test_data = split_data(X, y, train_split=0.7, val_split=0.15)
# 步骤4: 创建数据加载器
print("\n[4/7] 创建PyTorch数据加载器...")
train_loader, val_loader, test_loader = create_dataloaders(
train_data, val_data, test_data,
task='classification',
batch_size=64
)
print(f"✓ 训练批次:{len(train_loader)}")
print(f"✓ 验证批次:{len(val_loader)}")
print(f"✓ 测试批次:{len(test_loader)}")
# 步骤5: 创建模型
print("\n[5/7] 创建DLinear模型...")
model_config = DLinearConfig(
input_dim=X.shape[-1],
lookback=168,
task='classification',
hidden_dim=512,
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型已创建,可训练参数:{model.count_parameters():,} 个")
# 步骤6: 训练模型
print("\n[6/7] 开始训练模型...")
train_config = TrainingConfig(
batch_size=64,
max_epochs=50,
learning_rate=0.001,
early_stopping_patience=10,
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/dlinear_btc_1h_real'
)
trainer = Trainer(model, train_config, task='classification')
history = trainer.fit(train_loader, val_loader)
# 步骤7: 评估模型
print("\n[7/7] 在测试集上评估模型...")
metrics = trainer.evaluate(test_loader)
# 保存评估结果
results_path = './models/checkpoints/dlinear_btc_1h_real/test_metrics.json'
with open(results_path, 'w') as f:
json.dump(metrics, f, indent=2)
print(f"\n✓ 评估结果已保存至:{results_path}")
# 打印结果摘要
print("\n" + "="*80)
print("训练完成 - 最终测试结果")
print("="*80)
print(f"\n分类指标:")
print(f" 准确率: {metrics.get('accuracy', 0):.4f}")
print(f" F1分数: {metrics.get('f1', 0):.4f}")
print(f" 精确率: {metrics.get('precision', 0):.4f}")
print(f" 召回率: {metrics.get('recall', 0):.4f}")
if 'true_positives' in metrics:
print(f"\n混淆矩阵:")
print(f" 真阳性: {metrics['true_positives']}")
print(f" 真阴性: {metrics['true_negatives']}")
print(f" 假阳性: {metrics['false_positives']}")
print(f" 假阴性: {metrics['false_negatives']}")
print(f"\n回测结果:")
print(f" 总收益率: {metrics.get('backtest_total_return', 0)*100:.2f}%")
print(f" 最终资金: ${metrics.get('backtest_final_capital', 0):,.2f}")
print(f" 夏普比率: {metrics.get('backtest_sharpe_ratio', 0):.4f}")
print(f" 最大回撤: {metrics.get('backtest_max_drawdown', 0)*100:.2f}%")
print(f" 胜率: {metrics.get('backtest_win_rate', 0)*100:.2f}%")
print(f" 交易次数: {metrics.get('backtest_num_trades', 0)}")
# ONNX导出测试
print("\n" + "="*80)
print("测试ONNX导出")
print("="*80)
try:
import onnx
import onnxruntime as ort
import torch
import numpy as np
onnx_path = './models/checkpoints/dlinear_btc_1h_real/model.onnx'
print("\n[1/3] 导出模型到ONNX格式...")
model.to_onnx(onnx_path)
print(f"✓ ONNX模型已保存至:{onnx_path}")
print("\n[2/3] 验证ONNX模型...")
onnx_model = onnx.load(onnx_path)
onnx.checker.check_model(onnx_model)
print("✓ ONNX模型有效")
print("\n[3/3] 测试ONNX推理与PyTorch对比...")
test_input = torch.randn(1, 168, X.shape[-1])
model.cpu()
model.eval()
with torch.no_grad():
pytorch_output = model(test_input).numpy()
ort_session = ort.InferenceSession(onnx_path)
onnx_output = ort_session.run(None, {'input': test_input.numpy()})[0]
diff = np.abs(pytorch_output - onnx_output).max()
print(f"✓ PyTorch与ONNX的最大差异:{diff:.8f}")
if diff < 1e-5:
print("✓ ONNX导出成功!输出在容差范围内匹配。")
else:
print("⚠ 警告:ONNX输出与PyTorch有差异")
except Exception as e:
print(f"✗ ONNX导出失败:{e}")
print("\n" + "="*80)
print("✓ DLinear模型训练完成(真实数据)")
print("="*80)
if __name__ == "__main__":
main()
+258
View File
@@ -0,0 +1,258 @@
"""使用回归任务训练DLinear模型 - 预测未来1小时的实际价格变化百分比"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
import json
import numpy as np
def main():
print("="*80)
print("DLinear回归模型训练 - 预测未来1小时价格变化百分比")
print("="*80)
# 步骤1: 获取2020-2025年完整数据
print("\n[1/8] 从Binance获取BTC 2020-2025年完整数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2020-01-01',
end_date='2025-12-31',
use_cache=True
)
print(f"✓ 已获取 {len(df)} 行数据(约{len(df)/8760:.1f}年)")
print(f"✓ 时间范围:{df['timestamp'].min()} 至 {df['timestamp'].max()}")
# 分析数据覆盖的市场周期
print(f"\n✓ 价格变化分析:")
print(f" 起始价格: ${df['close'].iloc[0]:,.2f} (2020年)")
print(f" 结束价格: ${df['close'].iloc[-1]:,.2f} (2025年)")
total_return = (df['close'].iloc[-1] - df['close'].iloc[0]) / df['close'].iloc[0]
print(f" 总收益率: {total_return*100:+.2f}%")
# 按年分析
df['year'] = df['timestamp'].dt.year
print(f"\n✓ 年度表现分析(确保包含牛熊周期):")
for year in sorted(df['year'].unique()):
year_df = df[df['year'] == year]
if len(year_df) > 100:
year_return = (year_df['close'].iloc[-1] - year_df['close'].iloc[0]) / year_df['close'].iloc[0]
trend = "🚀 牛市" if year_return > 0.3 else ("🔴 熊市" if year_return < -0.3 else "📊 震荡")
print(f" {year}年: {year_return*100:+7.2f}% {trend}")
# 步骤2: 数据预处理(回归任务)
print("\n[2/8] 数据预处理(回归任务 - 预测未来1小时的收益率)...")
preprocessor = CryptoPreprocessor(
lookback=168, # 输入:过去168小时(7天)
forecast_horizon=1, # 预测:未来1小时
price_change_threshold=0.0 # 回归任务不使用阈值
)
X, y, processed_df = preprocessor.preprocess_pipeline(df, task='regression', fit=True)
print(f"✓ 已创建序列:X形状 = {X.shape}, y形状 = {y.shape}")
print(f"✓ 特征数量:{X.shape[-1]}")
# 分析标签分布(收益率)
print(f"\n✓ 标签统计(收益率):")
print(f" 最小值: {y.min()*100:.4f}%")
print(f" 最大值: {y.max()*100:.4f}%")
print(f" 均值: {y.mean()*100:.4f}%")
print(f" 标准差: {y.std()*100:.4f}%")
print(f" 中位数: {np.median(y)*100:.4f}%")
# 方向统计
n_positive = (y > 0).sum()
n_negative = (y <= 0).sum()
print(f"\n✓ 方向分布:")
print(f" 正收益(上涨): {n_positive} ({n_positive/len(y)*100:.1f}%)")
print(f" 负收益(下跌): {n_negative} ({n_negative/len(y)*100:.1f}%)")
# 步骤3: 划分数据集
print("\n[3/8] 划分训练/验证/测试集 (70/15/15)...")
train_data, val_data, test_data = split_data(X, y, train_split=0.7, val_split=0.15)
print(f" 训练集:{len(train_data[0])} 样本")
print(f" 收益率范围: {train_data[1].min()*100:.4f}% ~ {train_data[1].max()*100:.4f}%")
print(f" 均值: {train_data[1].mean()*100:.4f}%")
print(f" 验证集:{len(val_data[0])} 样本")
print(f" 收益率范围: {val_data[1].min()*100:.4f}% ~ {val_data[1].max()*100:.4f}%")
print(f" 均值: {val_data[1].mean()*100:.4f}%")
print(f" 测试集:{len(test_data[0])} 样本")
print(f" 收益率范围: {test_data[1].min()*100:.4f}% ~ {test_data[1].max()*100:.4f}%")
print(f" 均值: {test_data[1].mean()*100:.4f}%")
# 分析各数据集对应的时间范围
lookback = 168
train_end_idx = int(len(processed_df) * 0.7) - lookback
val_end_idx = int(len(processed_df) * 0.85) - lookback
train_df = processed_df.iloc[lookback:train_end_idx+lookback]
val_df = processed_df.iloc[train_end_idx+lookback:val_end_idx+lookback]
test_df = processed_df.iloc[val_end_idx+lookback:]
print(f"\n 训练集时间范围: {train_df['timestamp'].min()} 至 {train_df['timestamp'].max()}")
train_return = (train_df['close'].iloc[-1] - train_df['close'].iloc[0]) / train_df['close'].iloc[0]
print(f" 价格: ${train_df['close'].iloc[0]:,.0f} → ${train_df['close'].iloc[-1]:,.0f} ({train_return*100:+.1f}%)")
print(f" 验证集时间范围: {val_df['timestamp'].min()} 至 {val_df['timestamp'].max()}")
val_return = (val_df['close'].iloc[-1] - val_df['close'].iloc[0]) / val_df['close'].iloc[0]
print(f" 价格: ${val_df['close'].iloc[0]:,.0f} → ${val_df['close'].iloc[-1]:,.0f} ({val_return*100:+.1f}%)")
print(f" 测试集时间范围: {test_df['timestamp'].min()} 至 {test_df['timestamp'].max()}")
test_return = (test_df['close'].iloc[-1] - test_df['close'].iloc[0]) / test_df['close'].iloc[0]
print(f" 价格: ${test_df['close'].iloc[0]:,.0f} → ${test_df['close'].iloc[-1]:,.0f} ({test_return*100:+.1f}%)")
# 步骤4: 创建数据加载器
print("\n[4/8] 创建PyTorch数据加载器...")
train_loader, val_loader, test_loader = create_dataloaders(
train_data, val_data, test_data,
task='regression', # 回归任务
batch_size=64
)
# 步骤5: 创建模型
print("\n[5/8] 创建DLinear回归模型...")
model_config = DLinearConfig(
input_dim=X.shape[-1],
lookback=168,
task='regression', # 回归任务
hidden_dim=512,
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型已创建,可训练参数:{model.count_parameters():,} 个")
print(f"✓ 任务类型:回归(预测收益率百分比)")
print(f"✓ 损失函数:MSELoss")
# 步骤6: 训练模型
print("\n[6/8] 开始训练回归模型...")
train_config = TrainingConfig(
batch_size=64,
max_epochs=50,
learning_rate=0.001,
early_stopping_patience=10,
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/dlinear_btc_regression'
)
trainer = Trainer(
model,
train_config,
task='regression' # 回归任务,不需要pos_weight
)
print(f"✓ 训练数据规模:{len(train_data[0]):,} 样本")
history = trainer.fit(train_loader, val_loader)
# 步骤7: 评估模型
print("\n[7/8] 在测试集上评估模型...")
metrics = trainer.evaluate(test_loader)
# 保存评估结果
results_path = './models/checkpoints/dlinear_btc_regression/test_metrics.json'
with open(results_path, 'w') as f:
json.dump(metrics, f, indent=2)
print(f"\n✓ 评估结果已保存至:{results_path}")
# 步骤8: 结果分析
print("\n[8/8] 回归模型性能分析...")
print("\n" + "="*80)
print("训练完成 - 回归模型测试结果")
print("="*80)
print(f"\n📊 回归指标:")
print(f" RMSE (均方根误差): {metrics.get('rmse', 0):.6f} ({metrics.get('rmse', 0)*100:.4f}%)")
print(f" MAE (平均绝对误差): {metrics.get('mae', 0):.6f} ({metrics.get('mae', 0)*100:.4f}%)")
print(f" MAPE (平均绝对百分比误差): {metrics.get('mape', 0):.2f}%")
print(f" 方向准确率: {metrics.get('directional_accuracy', 0)*100:.2f}%")
print(f"\n💰 回测结果:")
print(f" 总收益率: {metrics.get('backtest_total_return', 0)*100:.2f}%")
print(f" 最终资金: ${metrics.get('backtest_final_capital', 0):,.2f}")
print(f" 夏普比率: {metrics.get('backtest_sharpe_ratio', 0):.4f}")
max_dd = metrics.get('backtest_max_drawdown', 0)
if not np.isnan(max_dd):
print(f" 最大回撤: {max_dd*100:.2f}%")
else:
print(f" 最大回撤: N/A")
print(f" 胜率: {metrics.get('backtest_win_rate', 0)*100:.2f}%")
print(f" 交易次数: {metrics.get('backtest_num_trades', 0)}")
# 与分类任务对比
print("\n" + "="*80)
print("📊 回归 vs 分类任务对比")
print("="*80)
print(f"\n二分类模型(之前的结果):")
print(f" 准确率: 49.76%")
print(f" F1分数: 0.3898")
print(f" 预测偏差: 68.1%预测下跌(过度悲观)")
print(f" 回测收益: -3.29%")
print(f"\n回归模型(当前结果):")
print(f" 方向准确率: {metrics.get('directional_accuracy', 0)*100:.2f}%")
print(f" RMSE: {metrics.get('rmse', 0)*100:.4f}%")
print(f" MAE: {metrics.get('mae', 0)*100:.4f}%")
print(f" 回测收益: {metrics.get('backtest_total_return', 0)*100:.2f}%")
# 性能评估
print(f"\n🎯 回归模型性能评估:")
dir_acc = metrics.get('directional_accuracy', 0)
if dir_acc > 0.55:
print(f" ✅ 方向准确率 {dir_acc*100:.1f}% > 55%,模型有预测能力")
elif dir_acc > 0.50:
print(f" ⚠️ 方向准确率 {dir_acc*100:.1f}% 略优于随机,预测能力较弱")
else:
print(f" ❌ 方向准确率 {dir_acc*100:.1f}% ≤ 50%,方向预测能力不足")
rmse = metrics.get('rmse', 0)
mae = metrics.get('mae', 0)
print(f"\n 误差分析:")
print(f" RMSE = {rmse*100:.4f}% - 平均预测误差的平方根")
print(f" MAE = {mae*100:.4f}% - 平均绝对误差")
if rmse < 0.01: # 小于1%
print(f" ✅ RMSE < 1%,预测精度较高")
elif rmse < 0.02: # 小于2%
print(f" ⚠️ RMSE在1-2%之间,预测精度中等")
else:
print(f" ❌ RMSE > 2%,预测误差较大")
total_return = metrics.get('backtest_total_return', 0)
if total_return > 0.05:
print(f"\n ✅ 回测收益 {total_return*100:.2f}% > 5%,策略有盈利潜力")
elif total_return > 0:
print(f"\n ⚠️ 回测收益 {total_return*100:.2f}% > 0,策略微弱盈利")
else:
print(f"\n ❌ 回测收益 {total_return*100:.2f}% ≤ 0,策略亏损")
print("\n💡 回归任务的优势:")
print(" ✅ 提供预测幅度信息(可用于动态仓位管理)")
print(" ✅ 评估风险收益比")
print(" ✅ 更适合量化交易策略")
print("\n💡 回归任务的挑战:")
print(" ❌ 预测具体数值比预测方向更难")
print(" ❌ 对异常值敏感")
print(" ❌ 需要更多训练数据才能稳定")
print("\n" + "="*80)
print("✓ 回归模型训练完成!")
print("="*80)
if __name__ == "__main__":
main()
+233
View File
@@ -0,0 +1,233 @@
"""方案A:使用2020-2025年BTC全部数据训练 - 准备实盘使用"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from data.fetcher import BinanceDataFetcher
from data.preprocessor import CryptoPreprocessor, split_data
from data.dataset import create_dataloaders
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
import json
import numpy as np
def main():
print("="*80)
print("方案A:BTC全量数据训练 (2020-2025)")
print("训练集:95% | 验证集:5%(仅用于early stopping)| 测试集:无")
print("="*80)
# ========== 步骤1: 获取全部BTC数据 ==========
print("\n[1/7] 获取BTC 2020-2025年全部数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2020-01-01',
end_date='2025-12-31',
use_cache=True
)
print(f"✓ 已获取 {len(df):,} 行数据(约{len(df)/8760:.1f}年)")
print(f"✓ 时间范围:{df['timestamp'].min()} 至 {df['timestamp'].max()}")
print(f"✓ 价格范围:${df['close'].iloc[0]:,.2f} → ${df['close'].iloc[-1]:,.2f}")
total_return = (df['close'].iloc[-1] - df['close'].iloc[0]) / df['close'].iloc[0]
print(f"✓ 总收益率:{total_return*100:+.2f}%")
# 市场周期分析
df['year'] = df['timestamp'].dt.year
print(f"\n✓ 市场周期覆盖:")
for year in sorted(df['year'].unique()):
year_df = df[df['year'] == year]
if len(year_df) > 100:
year_return = (year_df['close'].iloc[-1] - year_df['close'].iloc[0]) / year_df['close'].iloc[0]
trend = "🚀 牛市" if year_return > 0.3 else ("🔴 熊市" if year_return < -0.3 else "📊 震荡")
print(f" {year}年: {year_return*100:+7.2f}% {trend}")
# ========== 步骤2: 数据预处理 ==========
print("\n[2/7] 数据预处理(回归任务)...")
preprocessor = CryptoPreprocessor(
lookback=168, # 输入:过去7天
forecast_horizon=1, # 预测:未来1小时
price_change_threshold=0.0
)
X, y, processed_df = preprocessor.preprocess_pipeline(df, task='regression', fit=True)
print(f"✓ 序列数量:{len(X):,}")
print(f"✓ 序列形状:{X.shape}")
print(f"✓ 特征数量:{X.shape[-1]}")
print(f"\n✓ 标签统计(收益率):")
print(f" 最小值: {y.min()*100:.4f}%")
print(f" 最大值: {y.max()*100:.4f}%")
print(f" 均值: {y.mean()*100:.4f}%")
print(f" 标准差: {y.std()*100:.4f}%")
print(f" 中位数: {np.median(y)*100:.4f}%")
n_positive = (y > 0).sum()
n_negative = (y <= 0).sum()
print(f"\n✓ 方向分布:")
print(f" 上涨: {n_positive:,} ({n_positive/len(y)*100:.1f}%)")
print(f" 下跌: {n_negative:,} ({n_negative/len(y)*100:.1f}%)")
# ========== 步骤3: 划分数据集(95% 训练 + 5% 验证)==========
print("\n[3/7] 划分数据集(95%训练 + 5%验证)...")
print("⚠️ 注意:此次训练使用几乎全部数据,无独立测试集")
print(" 验证集仅用于early stopping防止过拟合")
n = len(X)
train_end = int(n * 0.95)
X_train = X[:train_end]
y_train = y[:train_end]
X_val = X[train_end:]
y_val = y[train_end:]
print(f"\n 训练集:{len(X_train):,} 样本 ({len(X_train)/n*100:.1f}%)")
print(f" 收益率范围: {y_train.min()*100:.4f}% ~ {y_train.max()*100:.4f}%")
print(f" 验证集:{len(X_val):,} 样本 ({len(X_val)/n*100:.1f}%)")
print(f" 收益率范围: {y_val.min()*100:.4f}% ~ {y_val.max()*100:.4f}%")
# 时间范围分析
lookback = 168
train_df = processed_df.iloc[lookback:train_end+lookback]
val_df = processed_df.iloc[train_end+lookback:]
print(f"\n 时间范围:")
print(f" 训练集: {train_df['timestamp'].min()} 至 {train_df['timestamp'].max()}")
train_return = (train_df['close'].iloc[-1] - train_df['close'].iloc[0]) / train_df['close'].iloc[0]
print(f" ${train_df['close'].iloc[0]:,.0f} → ${train_df['close'].iloc[-1]:,.0f} ({train_return*100:+.1f}%)")
print(f" 验证集: {val_df['timestamp'].min()} 至 {val_df['timestamp'].max()}")
val_return = (val_df['close'].iloc[-1] - val_df['close'].iloc[0]) / val_df['close'].iloc[0]
print(f" ${val_df['close'].iloc[0]:,.0f} → ${val_df['close'].iloc[-1]:,.0f} ({val_return*100:+.1f}%)")
# ========== 步骤4: 创建数据加载器 ==========
print("\n[4/7] 创建数据加载器...")
train_loader, val_loader, _ = create_dataloaders(
(X_train, y_train),
(X_val, y_val),
(X_val, y_val), # 测试集用验证集替代(不使用)
task='regression',
batch_size=64
)
# ========== 步骤5: 创建模型 ==========
print("\n[5/7] 创建DLinear回归模型...")
model_config = DLinearConfig(
input_dim=X.shape[-1],
lookback=168,
task='regression',
hidden_dim=512,
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型参数:{model.count_parameters():,} 个")
print(f"✓ 任务类型:回归(预测未来1小时收益率)")
print(f"✓ 损失函数:MSELoss")
# ========== 步骤6: 训练模型 ==========
print("\n[6/7] 开始全量数据训练...")
print(f"✓ 训练样本:{len(X_train):,} (使用了{len(X_train)/len(X)*100:.1f}%的数据)")
print(f"✓ 对比之前:之前训练集36,645样本,现在{len(X_train):,}样本(+{(len(X_train)-36645)/36645*100:.1f}%)")
train_config = TrainingConfig(
batch_size=64,
max_epochs=50,
learning_rate=0.001,
early_stopping_patience=15, # 增加patience,允许更充分训练
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/dlinear_btc_full_data'
)
trainer = Trainer(model, train_config, task='regression')
history = trainer.fit(train_loader, val_loader)
# ========== 步骤7: 保存最终模型 ==========
print("\n[7/7] 保存最终模型...")
# 在验证集上快速评估
print("\n在验证集上进行sanity check...")
val_metrics = trainer.evaluate(val_loader)
results_path = './models/checkpoints/dlinear_btc_full_data/final_metrics.json'
with open(results_path, 'w') as f:
json.dump(val_metrics, f, indent=2)
print(f"✓ 结果保存至:{results_path}")
# 同时保存预处理器(实盘使用时需要)
import pickle
preprocessor_path = './models/checkpoints/dlinear_btc_full_data/preprocessor.pkl'
with open(preprocessor_path, 'wb') as f:
pickle.dump(preprocessor, f)
print(f"✓ 预处理器保存至:{preprocessor_path}")
# ========== 最终报告 ==========
print("\n" + "="*80)
print("全量数据训练完成 - 最终报告")
print("="*80)
print(f"\n📊 训练概况:")
print(f" 数据时间范围:2020-01-01 至 2025-12-31 (6年)")
print(f" 训练样本数量:{len(X_train):,} (95%)")
print(f" 验证样本数量:{len(X_val):,} (5%)")
print(f" 特征数量:{X.shape[-1]}")
print(f" 模型参数:{model.count_parameters():,}")
print(f"\n📈 验证集性能(仅供参考,非独立测试):")
print(f" RMSE: {val_metrics.get('rmse', 0)*100:.4f}%")
print(f" MAE: {val_metrics.get('mae', 0)*100:.4f}%")
print(f" 方向准确率: {val_metrics.get('directional_accuracy', 0)*100:.2f}%")
print(f"\n💰 验证集回测(仅供参考):")
print(f" 总收益率: {val_metrics.get('backtest_total_return', 0)*100:.2f}%")
print(f" 夏普比率: {val_metrics.get('backtest_sharpe_ratio', 0):.4f}")
max_dd = val_metrics.get('backtest_max_drawdown', 0)
if not np.isnan(max_dd):
print(f" 最大回撤: {max_dd*100:.2f}%")
print(f" 胜率: {val_metrics.get('backtest_win_rate', 0)*100:.2f}%")
print("\n" + "="*80)
print("🎯 全量训练 vs 之前训练对比")
print("="*80)
print("\n之前训练(70/15/15分割):")
print(" 训练样本: 36,645")
print(" 时间范围: 2020-01 至 2024-03")
print(" 测试集方向准确率: 50.67%")
print(" 测试集回测收益: -16.71%")
print(f"\n当前全量训练(95/5分割):")
print(f" 训练样本: {len(X_train):,} (+{(len(X_train)-36645)/36645*100:.1f}%)")
print(f" 时间范围: 2020-01 至 {train_df['timestamp'].max().strftime('%Y-%m')}")
print(f" 验证集方向准确率: {val_metrics.get('directional_accuracy', 0)*100:.2f}%")
print(f" 验证集回测收益: {val_metrics.get('backtest_total_return', 0)*100:.2f}%")
print("\n" + "="*80)
print("⚠️ 重要说明")
print("="*80)
print("\n1. 此模型使用了95%的历史数据训练")
print("2. 验证集仅占5%,性能指标仅供参考,不代表真实泛化能力")
print("3. 模型已针对2020-2025年市场环境充分优化")
print("4. 适用场景:")
print(" ✅ 用于2026年未来数据的预测")
print(" ✅ 用于实盘交易(假设未来市场特征与历史相似)")
print(" ❌ 不适合评估在未见过数据上的泛化性能")
print("\n💡 下一步建议:")
print(" 1. 使用此模型预测2026年1月的BTC价格")
print(" 2. 在实盘中小仓位测试策略")
print(" 3. 定期(每月/季度)用新数据重新训练模型")
print(" 4. 监控实盘性能,如果偏差大说明市场环境变化")
print("\n" + "="*80)
print("✓ 训练完成!模型已保存到 ./models/checkpoints/dlinear_btc_full_data/")
print("="*80)
if __name__ == "__main__":
main()
+260
View File
@@ -0,0 +1,260 @@
"""直接预测未来1小时的价格(而非收益率)"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
import pandas as pd
import pandas_ta as ta
from sklearn.preprocessing import StandardScaler
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from data.fetcher import BinanceDataFetcher
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
import json
import pickle
class PricePreprocessor:
"""预测未来价格的预处理器"""
def __init__(self, lookback=168, forecast_horizon=1):
self.lookback = lookback
self.forecast_horizon = forecast_horizon
self.scaler = StandardScaler()
self.price_scaler = StandardScaler() # 单独的价格缩放器
self.is_fitted = False
def add_features(self, df):
"""添加技术指标"""
df = df.copy()
# RSI
df['rsi'] = ta.rsi(df['close'], length=14)
# MACD
macd = ta.macd(df['close'])
if macd is not None and not macd.empty:
df['macd'] = macd.iloc[:, 0]
df['macd_hist'] = macd.iloc[:, 1]
df['macd_signal'] = macd.iloc[:, 2]
return df
def create_labels(self, df):
"""创建标签:未来1小时的价格(绝对价格)"""
df = df.copy()
# 标签是未来1小时的收盘价
df['future_price'] = df['close'].shift(-self.forecast_horizon)
df['label'] = df['future_price']
return df[:-self.forecast_horizon]
def preprocess(self, df, fit=True):
"""完整预处理流程"""
# 添加技术指标
df = self.add_features(df)
df = self.create_labels(df)
# 删除NaN
df = df.dropna().reset_index(drop=True)
# 特征列
feature_cols = ['open', 'high', 'low', 'close', 'volume',
'rsi', 'macd', 'macd_hist', 'macd_signal']
# 标准化特征
if fit:
scaled_features = self.scaler.fit_transform(df[feature_cols])
# 单独标准化价格标签(用于训练)
scaled_labels = self.price_scaler.fit_transform(df[['label']])
self.is_fitted = True
self.feature_columns = feature_cols
else:
if not self.is_fitted:
raise ValueError("Must fit before transform")
scaled_features = self.scaler.transform(df[feature_cols])
scaled_labels = self.price_scaler.transform(df[['label']])
# 创建序列
X, y = [], []
for i in range(len(scaled_features) - self.lookback):
X.append(scaled_features[i:i + self.lookback])
y.append(scaled_labels[i + self.lookback, 0]) # 标准化后的价格
return np.array(X), np.array(y), df
def inverse_transform_price(self, scaled_price):
"""将标准化的价格还原为真实价格"""
return self.price_scaler.inverse_transform(scaled_price.reshape(-1, 1)).flatten()
def main():
print("="*80)
print("训练模型:直接预测未来1小时价格")
print("="*80)
# ========== 步骤1: 获取数据 ==========
print("\n[1/7] 获取BTC 2020-2025年数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2020-01-01',
end_date='2025-12-31',
use_cache=True
)
print(f"✓ 数据量: {len(df):,} 行")
print(f"✓ 时间: {df['timestamp'].min()} 至 {df['timestamp'].max()}")
print(f"✓ 价格: ${df['close'].iloc[0]:,.2f} → ${df['close'].iloc[-1]:,.2f}")
# ========== 步骤2: 预处理 ==========
print("\n[2/7] 数据预处理(预测绝对价格)...")
preprocessor = PricePreprocessor(lookback=168, forecast_horizon=1)
X, y, processed_df = preprocessor.preprocess(df, fit=True)
print(f"✓ 序列数量: {len(X):,}")
print(f"✓ 序列形状: {X.shape}")
print(f"✓ 特征数量: {len(preprocessor.feature_columns)}")
# 还原真实价格看看统计
y_real = preprocessor.inverse_transform_price(y)
print(f"\n✓ 目标价格统计(真实值):")
print(f" 最小值: ${y_real.min():,.2f}")
print(f" 最大值: ${y_real.max():,.2f}")
print(f" 均值: ${y_real.mean():,.2f}")
# ========== 步骤3: 划分数据集(95/5)==========
print("\n[3/7] 划分数据集(95%训练 + 5%验证)...")
n = len(X)
train_end = int(n * 0.95)
X_train, y_train = X[:train_end], y[:train_end]
X_val, y_val = X[train_end:], y[train_end:]
print(f" 训练集: {len(X_train):,} 样本")
print(f" 验证集: {len(X_val):,} 样本")
# ========== 步骤4: 创建数据加载器 ==========
print("\n[4/7] 创建数据加载器...")
train_dataset = TensorDataset(
torch.FloatTensor(X_train),
torch.FloatTensor(y_train)
)
val_dataset = TensorDataset(
torch.FloatTensor(X_val),
torch.FloatTensor(y_val)
)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=False)
val_loader = DataLoader(val_dataset, batch_size=64, shuffle=False)
# ========== 步骤5: 创建模型 ==========
print("\n[5/7] 创建DLinear模型...")
model_config = DLinearConfig(
input_dim=9,
lookback=168,
task='regression', # 仍然是回归,但预测价格而非收益率
hidden_dim=256,
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型参数: {model.count_parameters():,}")
print(f"✓ 预测目标: 未来1小时的绝对价格")
# ========== 步骤6: 训练 ==========
print("\n[6/7] 开始训练...")
train_config = TrainingConfig(
batch_size=64,
max_epochs=50,
learning_rate=0.001,
early_stopping_patience=15,
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/dlinear_predict_price'
)
trainer = Trainer(model, train_config, task='regression')
history = trainer.fit(train_loader, val_loader)
# ========== 步骤7: 评估 ==========
print("\n[7/7] 评估模型...")
# 在验证集上预测
device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu')
model = model.to(device)
model.eval()
val_preds = []
with torch.no_grad():
for batch_X, _ in val_loader:
batch_X = batch_X.to(device)
pred = model(batch_X).squeeze()
val_preds.extend(pred.cpu().numpy())
val_preds = np.array(val_preds)
# 还原为真实价格
pred_prices = preprocessor.inverse_transform_price(val_preds)
true_prices = preprocessor.inverse_transform_price(y_val)
# 计算当前价格(用于判断方向)
val_start_idx = train_end + 168 # +168是lookback
current_prices = processed_df['close'].iloc[val_start_idx:val_start_idx+len(pred_prices)].values
# 方向准确率
pred_direction = (pred_prices > current_prices).astype(int)
true_direction = (true_prices > current_prices).astype(int)
direction_acc = (pred_direction == true_direction).mean()
# 价格预测误差
price_mae = np.abs(pred_prices - true_prices).mean()
price_rmse = np.sqrt(((pred_prices - true_prices) ** 2).mean())
price_mape = np.mean(np.abs((true_prices - pred_prices) / true_prices)) * 100
print("\n" + "="*80)
print("训练完成 - 验证集性能")
print("="*80)
print(f"\n📊 价格预测误差:")
print(f" MAE: ${price_mae:,.2f}")
print(f" RMSE: ${price_rmse:,.2f}")
print(f" MAPE: {price_mape:.2f}%")
print(f"\n📈 方向预测:")
print(f" 方向准确率: {direction_acc*100:.2f}%")
print(f" 预测上涨: {(pred_direction==1).sum()} / 实际上涨: {(true_direction==1).sum()}")
print(f" 预测下跌: {(pred_direction==0).sum()} / 实际下跌: {(true_direction==0).sum()}")
print(f"\n💡 价格预测示例(前10个):")
print(f"{'当前价格':<12} {'预测价格':<12} {'实际价格':<12} {'预测方向':<8} {'实际方向':<8}")
print("-"*70)
for i in range(min(10, len(pred_prices))):
pred_dir = "上涨" if pred_prices[i] > current_prices[i] else "下跌"
true_dir = "上涨" if true_prices[i] > current_prices[i] else "下跌"
print(f"${current_prices[i]:<11,.2f} ${pred_prices[i]:<11,.2f} ${true_prices[i]:<11,.2f} "
f"{pred_dir:<8} {true_dir:<8}")
# 保存模型和预处理器
with open('./models/checkpoints/dlinear_predict_price/preprocessor.pkl', 'wb') as f:
pickle.dump(preprocessor, f)
# 保存评估指标
metrics = {
'price_mae': float(price_mae),
'price_rmse': float(price_rmse),
'price_mape': float(price_mape),
'direction_accuracy': float(direction_acc)
}
with open('./models/checkpoints/dlinear_predict_price/metrics.json', 'w') as f:
json.dump(metrics, f, indent=2)
print(f"\n✓ 模型保存至: ./models/checkpoints/dlinear_predict_price/")
print("="*80)
if __name__ == "__main__":
main()
+206
View File
@@ -0,0 +1,206 @@
"""简化特征训练:只使用OHLCV + RSI + MACD"""
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
import pandas as pd
import pandas_ta as ta
from sklearn.preprocessing import StandardScaler
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from data.fetcher import BinanceDataFetcher
from models.dlinear.model import DLinear, DLinearConfig
from training.trainer import Trainer
from training.config import TrainingConfig
import json
import pickle
class SimplePreprocessor:
"""简化的预处理器:只使用OHLCV + RSI + MACD"""
def __init__(self, lookback=168, forecast_horizon=1):
self.lookback = lookback
self.forecast_horizon = forecast_horizon
self.scaler = StandardScaler()
self.is_fitted = False
def add_features(self, df):
"""添加RSI和MACD特征"""
df = df.copy()
# RSI (14 periods)
df['rsi'] = ta.rsi(df['close'], length=14)
# MACD
macd = ta.macd(df['close'])
if macd is not None and not macd.empty:
df['macd'] = macd.iloc[:, 0]
df['macd_hist'] = macd.iloc[:, 1]
df['macd_signal'] = macd.iloc[:, 2]
return df
def create_labels(self, df):
"""创建标签:未来1小时的收益率"""
df = df.copy()
df['future_price'] = df['close'].shift(-self.forecast_horizon)
df['future_return'] = (df['future_price'] - df['close']) / df['close']
df['label'] = df['future_return']
return df[:-self.forecast_horizon]
def preprocess(self, df, fit=True):
"""完整预处理流程"""
# 添加技术指标
df = self.add_features(df)
df = self.create_labels(df)
# 删除NaN
df = df.dropna().reset_index(drop=True)
# 选择特征列:OHLCV + RSI + MACD
feature_cols = ['open', 'high', 'low', 'close', 'volume',
'rsi', 'macd', 'macd_hist', 'macd_signal']
# 标准化
if fit:
scaled = self.scaler.fit_transform(df[feature_cols])
self.is_fitted = True
self.feature_columns = feature_cols
else:
if not self.is_fitted:
raise ValueError("Must fit before transform")
scaled = self.scaler.transform(df[feature_cols])
# 创建序列
X, y = [], []
for i in range(len(scaled) - self.lookback):
X.append(scaled[i:i + self.lookback])
y.append(df['label'].iloc[i + self.lookback])
return np.array(X), np.array(y), df
def main():
print("="*80)
print("简化特征训练:OHLCV + RSI + MACD")
print("="*80)
# ========== 步骤1: 获取数据 ==========
print("\n[1/7] 获取BTC 2020-2025年数据...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv(
symbol='BTC',
timeframe='1h',
start_date='2020-01-01',
end_date='2025-12-31',
use_cache=True
)
print(f"✓ 数据量: {len(df):,} 行")
print(f"✓ 时间: {df['timestamp'].min()} 至 {df['timestamp'].max()}")
print(f"✓ 价格: ${df['close'].iloc[0]:,.2f} → ${df['close'].iloc[-1]:,.2f}")
# ========== 步骤2: 预处理 ==========
print("\n[2/7] 数据预处理(只使用9个特征)...")
preprocessor = SimplePreprocessor(lookback=168, forecast_horizon=1)
X, y, processed_df = preprocessor.preprocess(df, fit=True)
print(f"✓ 序列数量: {len(X):,}")
print(f"✓ 序列形状: {X.shape}")
print(f"✓ 特征列表:")
for i, col in enumerate(preprocessor.feature_columns, 1):
print(f" {i}. {col}")
print(f"\n✓ 标签统计:")
print(f" 收益率范围: {y.min()*100:.4f}% ~ {y.max()*100:.4f}%")
print(f" 均值: {y.mean()*100:.4f}%")
print(f" 上涨样本: {(y>0).sum()} ({(y>0).sum()/len(y)*100:.1f}%)")
print(f" 下跌样本: {(y<=0).sum()} ({(y<=0).sum()/len(y)*100:.1f}%)")
# ========== 步骤3: 划分数据集(95/5)==========
print("\n[3/7] 划分数据集(95%训练 + 5%验证)...")
n = len(X)
train_end = int(n * 0.95)
X_train, y_train = X[:train_end], y[:train_end]
X_val, y_val = X[train_end:], y[train_end:]
print(f" 训练集: {len(X_train):,} 样本")
print(f" 验证集: {len(X_val):,} 样本")
# ========== 步骤4: 创建数据加载器 ==========
print("\n[4/7] 创建数据加载器...")
train_dataset = TensorDataset(
torch.FloatTensor(X_train),
torch.FloatTensor(y_train)
)
val_dataset = TensorDataset(
torch.FloatTensor(X_val),
torch.FloatTensor(y_val)
)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=False)
val_loader = DataLoader(val_dataset, batch_size=64, shuffle=False)
# ========== 步骤5: 创建模型 ==========
print("\n[5/7] 创建DLinear模型...")
model_config = DLinearConfig(
input_dim=9, # OHLCV + RSI + MACD(3个)
lookback=168,
task='regression',
hidden_dim=256, # 减小隐藏层
dropout=0.1,
moving_avg_kernel=25
)
model = DLinear(model_config)
print(f"✓ 模型参数: {model.count_parameters():,}")
# ========== 步骤6: 训练 ==========
print("\n[6/7] 开始训练...")
train_config = TrainingConfig(
batch_size=64,
max_epochs=50,
learning_rate=0.001,
early_stopping_patience=15,
lr_scheduler='plateau',
save_best=True,
checkpoint_dir='./models/checkpoints/dlinear_simple_features'
)
trainer = Trainer(model, train_config, task='regression')
history = trainer.fit(train_loader, val_loader)
# ========== 步骤7: 评估并保存 ==========
print("\n[7/7] 评估并保存...")
val_metrics = trainer.evaluate(val_loader)
# 保存结果
with open('./models/checkpoints/dlinear_simple_features/metrics.json', 'w') as f:
json.dump(val_metrics, f, indent=2)
# 保存预处理器
with open('./models/checkpoints/dlinear_simple_features/preprocessor.pkl', 'wb') as f:
pickle.dump(preprocessor, f)
print("\n" + "="*80)
print("训练完成")
print("="*80)
print(f"\n📊 验证集性能:")
print(f" RMSE: {val_metrics.get('rmse', 0)*100:.4f}%")
print(f" MAE: {val_metrics.get('mae', 0)*100:.4f}%")
print(f" 方向准确率: {val_metrics.get('directional_accuracy', 0)*100:.2f}%")
print(f"\n💰 验证集回测:")
print(f" 总收益率: {val_metrics.get('backtest_total_return', 0)*100:.2f}%")
print(f" 夏普比率: {val_metrics.get('backtest_sharpe_ratio', 0):.4f}")
print(f"\n✓ 模型保存至: ./models/checkpoints/dlinear_simple_features/")
print(f"✓ 特征数量: 9 (OHLCV + RSI + MACD)")
print("="*80)
if __name__ == "__main__":
main()
+1
View File
@@ -0,0 +1 @@
"""Training and evaluation utilities."""
+58
View File
@@ -0,0 +1,58 @@
"""Training configuration."""
from dataclasses import dataclass
from typing import Literal
@dataclass
class TrainingConfig:
"""Configuration for model training."""
# Training parameters
batch_size: int = 64
max_epochs: int = 100
learning_rate: float = 0.001
weight_decay: float = 1e-5
# Early stopping
early_stopping_patience: int = 10
early_stopping_min_delta: float = 1e-4
# Learning rate scheduler
lr_scheduler: Literal['plateau', 'cosine', 'none'] = 'plateau'
lr_patience: int = 5
lr_factor: float = 0.5
# Device
device: str = 'auto' # 'auto', 'mps', 'cuda', or 'cpu'
# Logging
log_interval: int = 10 # Log every N batches
# Checkpointing
save_best: bool = True
checkpoint_dir: str = './models/checkpoints'
def get_device(self) -> str:
"""Get the device to use for training.
Returns:
Device string ('mps', 'cuda', or 'cpu')
"""
import torch
if self.device != 'auto':
return self.device
# Auto-detect best available device
if torch.backends.mps.is_available():
return 'mps'
elif torch.cuda.is_available():
return 'cuda'
else:
return 'cpu'
if __name__ == "__main__":
config = TrainingConfig()
print(f"Using device: {config.get_device()}")
+296
View File
@@ -0,0 +1,296 @@
"""Model evaluation metrics and utilities."""
from typing import Dict, Literal
import numpy as np
import torch
from sklearn.metrics import (
accuracy_score,
precision_score,
recall_score,
f1_score,
confusion_matrix,
mean_squared_error,
mean_absolute_error
)
class ModelEvaluator:
"""Evaluate model performance with task-specific metrics."""
def __init__(self, task: Literal['classification', 'regression']):
"""Initialize evaluator.
Args:
task: Task type ('classification' or 'regression')
"""
self.task = task
def evaluate(
self,
y_true: np.ndarray,
y_pred: np.ndarray,
y_prob: np.ndarray = None
) -> Dict[str, float]:
"""Compute evaluation metrics.
Args:
y_true: True labels
y_pred: Predicted labels/values
y_prob: Prediction probabilities (for classification)
Returns:
Dictionary of metrics
"""
if self.task == 'classification':
return self._evaluate_classification(y_true, y_pred, y_prob)
else:
return self._evaluate_regression(y_true, y_pred)
def _evaluate_classification(
self,
y_true: np.ndarray,
y_pred: np.ndarray,
y_prob: np.ndarray = None
) -> Dict[str, float]:
"""Compute classification metrics.
Args:
y_true: True labels (0 or 1)
y_pred: Predicted labels (0 or 1)
y_prob: Prediction probabilities
Returns:
Dictionary of classification metrics
"""
metrics = {
'accuracy': accuracy_score(y_true, y_pred),
'precision': precision_score(y_true, y_pred, zero_division=0),
'recall': recall_score(y_true, y_pred, zero_division=0),
'f1': f1_score(y_true, y_pred, zero_division=0)
}
# Confusion matrix
cm = confusion_matrix(y_true, y_pred)
if cm.shape == (2, 2):
tn, fp, fn, tp = cm.ravel()
metrics['true_negatives'] = int(tn)
metrics['false_positives'] = int(fp)
metrics['false_negatives'] = int(fn)
metrics['true_positives'] = int(tp)
# Specificity
if (tn + fp) > 0:
metrics['specificity'] = tn / (tn + fp)
# Directional accuracy (if probabilities provided)
if y_prob is not None:
metrics['mean_confidence'] = float(np.mean(y_prob))
return metrics
def _evaluate_regression(
self,
y_true: np.ndarray,
y_pred: np.ndarray
) -> Dict[str, float]:
"""Compute regression metrics.
Args:
y_true: True values
y_pred: Predicted values
Returns:
Dictionary of regression metrics
"""
# Basic regression metrics
mse = mean_squared_error(y_true, y_pred)
mae = mean_absolute_error(y_true, y_pred)
rmse = np.sqrt(mse)
# MAPE (Mean Absolute Percentage Error)
# Avoid division by zero
epsilon = 1e-8
mape = np.mean(np.abs((y_true - y_pred) / (np.abs(y_true) + epsilon))) * 100
# Directional accuracy (sign agreement)
directional_accuracy = np.mean(np.sign(y_true) == np.sign(y_pred))
metrics = {
'mse': float(mse),
'rmse': float(rmse),
'mae': float(mae),
'mape': float(mape),
'directional_accuracy': float(directional_accuracy)
}
return metrics
def backtest_strategy(
self,
y_true: np.ndarray,
y_pred: np.ndarray,
initial_capital: float = 10000.0,
transaction_cost: float = 0.001
) -> Dict[str, float]:
"""Simple backtesting with long/short strategy.
Args:
y_true: True returns
y_pred: Predicted labels (classification) or returns (regression)
initial_capital: Starting capital
transaction_cost: Transaction cost per trade (e.g., 0.001 = 0.1%)
Returns:
Dictionary of backtest metrics
"""
capital = initial_capital
positions = []
returns_list = []
for i in range(len(y_pred)):
if self.task == 'classification':
# Classification: y_pred已经是0/1的标签(不是概率)
# 1 = buy, 0 = sell/short
position = 1 if y_pred[i] == 1 else -1
else:
# Regression: positive prediction = buy, negative = sell
position = 1 if y_pred[i] > 0 else -1
# Calculate return based on position
actual_return = y_true[i]
strategy_return = position * actual_return
# Apply transaction cost if position changes
if i > 0 and positions[-1] != position:
strategy_return -= transaction_cost
# Update capital
capital *= (1 + strategy_return)
positions.append(position)
returns_list.append(strategy_return)
# Calculate metrics
returns_array = np.array(returns_list)
total_return = (capital - initial_capital) / initial_capital
# Sharpe ratio (annualized, assuming hourly data)
# For hourly data: sqrt(24 * 365) ≈ 93
if len(returns_array) > 0 and returns_array.std() > 0:
sharpe_ratio = (returns_array.mean() / returns_array.std()) * np.sqrt(93)
else:
sharpe_ratio = 0.0
# Maximum drawdown
cumulative = np.cumprod(1 + returns_array)
running_max = np.maximum.accumulate(cumulative)
drawdown = (cumulative - running_max) / running_max
max_drawdown = float(drawdown.min())
# Win rate
win_rate = np.mean(returns_array > 0)
return {
'total_return': float(total_return),
'final_capital': float(capital),
'sharpe_ratio': float(sharpe_ratio),
'max_drawdown': float(max_drawdown),
'win_rate': float(win_rate),
'num_trades': int(np.sum(np.diff(positions) != 0))
}
def evaluate_model(
model: torch.nn.Module,
dataloader: torch.utils.data.DataLoader,
task: Literal['classification', 'regression'],
device: str = 'cpu'
) -> Dict[str, float]:
"""Evaluate a model on a dataset.
Args:
model: PyTorch model
dataloader: DataLoader with evaluation data
task: Task type
device: Device to run evaluation on
Returns:
Dictionary of evaluation metrics
"""
model.eval()
model = model.to(device)
all_preds = []
all_probs = []
all_labels = []
with torch.no_grad():
for batch_X, batch_y in dataloader:
batch_X = batch_X.to(device)
batch_y = batch_y.to(device)
# Forward pass
outputs = model(batch_X).squeeze()
if task == 'classification':
# 注意:修复后的模型输出logits,需要应用sigmoid得到概率
probs = torch.sigmoid(outputs).cpu().numpy()
preds = (probs > 0.5).astype(int)
# 确保是1维数组
if probs.ndim == 0:
probs = np.array([probs])
if preds.ndim == 0:
preds = np.array([preds])
all_probs.extend(probs.flatten())
all_preds.extend(preds.flatten())
else:
# Regression predictions
preds = outputs.cpu().numpy()
if preds.ndim == 0:
preds = np.array([preds])
all_preds.extend(preds.flatten())
# Labels
labels = batch_y.cpu().numpy()
if labels.ndim == 0:
labels = np.array([labels])
all_labels.extend(labels.flatten())
# Convert to arrays
y_true = np.array(all_labels)
y_pred = np.array(all_preds)
y_prob = np.array(all_probs) if task == 'classification' else None
# Evaluate
evaluator = ModelEvaluator(task)
metrics = evaluator.evaluate(y_true, y_pred, y_prob)
# Add backtesting metrics
backtest = evaluator.backtest_strategy(y_true, y_pred)
metrics.update({f'backtest_{k}': v for k, v in backtest.items()})
return metrics
if __name__ == "__main__":
# Test classification evaluation
y_true = np.array([0, 1, 1, 0, 1, 0, 1, 1, 0, 0])
y_pred = np.array([0, 1, 1, 0, 0, 1, 1, 1, 0, 1])
y_prob = np.array([0.3, 0.8, 0.9, 0.2, 0.4, 0.6, 0.7, 0.85, 0.1, 0.55])
evaluator = ModelEvaluator('classification')
metrics = evaluator.evaluate(y_true, y_pred, y_prob)
print("Classification metrics:")
for k, v in metrics.items():
print(f" {k}: {v}")
# Test regression evaluation
y_true_reg = np.random.randn(100) * 0.02
y_pred_reg = y_true_reg + np.random.randn(100) * 0.01
evaluator_reg = ModelEvaluator('regression')
metrics_reg = evaluator_reg.evaluate(y_true_reg, y_pred_reg)
print("\nRegression metrics:")
for k, v in metrics_reg.items():
print(f" {k}: {v}")
+361
View File
@@ -0,0 +1,361 @@
"""Training orchestration and utilities."""
import os
from pathlib import Path
from typing import Optional, Dict
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from tqdm import tqdm
import json
from .config import TrainingConfig
from .evaluator import evaluate_model
class EarlyStopping:
"""Early stopping to stop training when validation loss doesn't improve."""
def __init__(self, patience: int = 10, min_delta: float = 1e-4):
"""Initialize early stopping.
Args:
patience: Number of epochs to wait before stopping
min_delta: Minimum change to qualify as improvement
"""
self.patience = patience
self.min_delta = min_delta
self.counter = 0
self.best_loss = None
self.should_stop = False
def __call__(self, val_loss: float) -> bool:
"""Check if training should stop.
Args:
val_loss: Current validation loss
Returns:
True if training should stop
"""
if self.best_loss is None:
self.best_loss = val_loss
elif val_loss > self.best_loss - self.min_delta:
self.counter += 1
if self.counter >= self.patience:
self.should_stop = True
else:
self.best_loss = val_loss
self.counter = 0
return self.should_stop
class Trainer:
"""Model trainer with support for MPS/CUDA/CPU."""
def __init__(
self,
model: nn.Module,
config: TrainingConfig,
task: str = 'classification',
pos_weight: float = None # 添加类别权重参数
):
"""Initialize trainer.
Args:
model: PyTorch model to train
config: Training configuration
task: Task type ('classification' or 'regression')
pos_weight: Positive class weight for imbalanced data
"""
self.model = model
self.config = config
self.task = task
self.pos_weight = pos_weight
# Set device
self.device = config.get_device()
self.model = self.model.to(self.device)
print(f"Training on device: {self.device}")
# Loss function (with class weight if provided)
if hasattr(model, 'get_loss_function'):
self.criterion = model.get_loss_function(pos_weight=pos_weight, device=self.device)
else:
if task == 'classification':
if pos_weight is not None:
# 使用float32以兼容MPS
pos_weight_tensor = torch.tensor([pos_weight], dtype=torch.float32).to(self.device)
self.criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight_tensor)
else:
self.criterion = nn.BCELoss()
else:
self.criterion = nn.MSELoss()
if pos_weight is not None:
print(f"Using weighted loss - positive class weight: {pos_weight:.2f}")
# Optimizer
self.optimizer = torch.optim.AdamW(
model.parameters(),
lr=config.learning_rate,
weight_decay=config.weight_decay
)
# Learning rate scheduler
if config.lr_scheduler == 'plateau':
self.scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
self.optimizer,
mode='min',
factor=config.lr_factor,
patience=config.lr_patience,
verbose=True
)
elif config.lr_scheduler == 'cosine':
self.scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
self.optimizer,
T_max=config.max_epochs
)
else:
self.scheduler = None
# Early stopping
self.early_stopping = EarlyStopping(
patience=config.early_stopping_patience,
min_delta=config.early_stopping_min_delta
)
# Training history
self.history = {
'train_loss': [],
'val_loss': [],
'learning_rates': []
}
# Best model tracking
self.best_val_loss = float('inf')
self.best_model_path = None
def train_epoch(self, train_loader: DataLoader) -> float:
"""Train for one epoch.
Args:
train_loader: Training data loader
Returns:
Average training loss
"""
self.model.train()
total_loss = 0.0
num_batches = 0
pbar = tqdm(train_loader, desc='Training')
for batch_X, batch_y in pbar:
batch_X = batch_X.to(self.device)
batch_y = batch_y.to(self.device)
# Forward pass
self.optimizer.zero_grad()
outputs = self.model(batch_X).squeeze()
# Handle different output shapes
if self.task == 'classification':
batch_y = batch_y.float()
loss = self.criterion(outputs, batch_y)
# Backward pass
loss.backward()
self.optimizer.step()
# Track loss
total_loss += loss.item()
num_batches += 1
# Update progress bar
pbar.set_postfix({'loss': f'{loss.item():.4f}'})
return total_loss / num_batches
def validate(self, val_loader: DataLoader) -> float:
"""Validate the model.
Args:
val_loader: Validation data loader
Returns:
Average validation loss
"""
self.model.eval()
total_loss = 0.0
num_batches = 0
with torch.no_grad():
for batch_X, batch_y in val_loader:
batch_X = batch_X.to(self.device)
batch_y = batch_y.to(self.device)
outputs = self.model(batch_X).squeeze()
if self.task == 'classification':
batch_y = batch_y.float()
loss = self.criterion(outputs, batch_y)
total_loss += loss.item()
num_batches += 1
return total_loss / num_batches
def fit(
self,
train_loader: DataLoader,
val_loader: DataLoader,
save_dir: Optional[str] = None
) -> Dict:
"""Train the model.
Args:
train_loader: Training data loader
val_loader: Validation data loader
save_dir: Directory to save checkpoints
Returns:
Training history
"""
if save_dir is None:
save_dir = self.config.checkpoint_dir
save_dir = Path(save_dir)
save_dir.mkdir(parents=True, exist_ok=True)
print(f"Starting training for {self.config.max_epochs} epochs...")
print(f"Model parameters: {self.model.count_parameters()}")
for epoch in range(self.config.max_epochs):
# Train
train_loss = self.train_epoch(train_loader)
# Validate
val_loss = self.validate(val_loader)
# Update history
self.history['train_loss'].append(train_loss)
self.history['val_loss'].append(val_loss)
current_lr = self.optimizer.param_groups[0]['lr']
self.history['learning_rates'].append(current_lr)
# Print progress
print(f"Epoch {epoch + 1}/{self.config.max_epochs}")
print(f" Train loss: {train_loss:.4f}")
print(f" Val loss: {val_loss:.4f}")
print(f" LR: {current_lr:.6f}")
# Save best model
if val_loss < self.best_val_loss:
self.best_val_loss = val_loss
if self.config.save_best:
self.best_model_path = save_dir / 'best_model.pt'
self.model.save(str(self.best_model_path))
print(f" Saved best model (val_loss: {val_loss:.4f})")
# Learning rate scheduling
if self.scheduler is not None:
if self.config.lr_scheduler == 'plateau':
self.scheduler.step(val_loss)
else:
self.scheduler.step()
# Early stopping
if self.early_stopping(val_loss):
print(f"Early stopping at epoch {epoch + 1}")
break
# Save final model
final_model_path = save_dir / 'final_model.pt'
self.model.save(str(final_model_path))
# Save training history
history_path = save_dir / 'training_history.json'
with open(history_path, 'w') as f:
json.dump(self.history, f, indent=2)
print(f"\nTraining completed!")
print(f"Best validation loss: {self.best_val_loss:.4f}")
print(f"Best model saved to: {self.best_model_path}")
return self.history
def evaluate(self, test_loader: DataLoader) -> Dict[str, float]:
"""Evaluate the model on test data.
Args:
test_loader: Test data loader
Returns:
Dictionary of evaluation metrics
"""
# Load best model if available
if self.best_model_path is not None and self.best_model_path.exists():
print(f"Loading best model from {self.best_model_path}")
checkpoint = torch.load(self.best_model_path, map_location=self.device)
self.model.load_state_dict(checkpoint['model_state_dict'])
# Evaluate
metrics = evaluate_model(self.model, test_loader, self.task, self.device)
print("\nTest Metrics:")
for key, value in metrics.items():
print(f" {key}: {value:.4f}")
return metrics
if __name__ == "__main__":
# Test the trainer
from ..models.dlinear.model import DLinear, DLinearConfig
from ..data.fetcher import BinanceDataFetcher
from ..data.preprocessor import CryptoPreprocessor, split_data
from ..data.dataset import create_dataloaders
# Fetch data
print("Fetching data...")
fetcher = BinanceDataFetcher()
df = fetcher.fetch_ohlcv('BTC', '1h')
# Preprocess
print("\nPreprocessing...")
preprocessor = CryptoPreprocessor(lookback=168)
X, y, _ = preprocessor.preprocess_pipeline(df, task='classification')
# Split
train_data, val_data, test_data = split_data(X, y)
# Create dataloaders
train_loader, val_loader, test_loader = create_dataloaders(
train_data, val_data, test_data,
batch_size=32
)
# Create model
print("\nCreating model...")
config = DLinearConfig(
input_dim=X.shape[-1],
lookback=168,
task='classification'
)
model = DLinear(config)
# Train
train_config = TrainingConfig(
batch_size=32,
max_epochs=5, # Short test
early_stopping_patience=3
)
trainer = Trainer(model, train_config, task='classification')
history = trainer.fit(train_loader, val_loader, save_dir='/tmp/test_training')
# Evaluate
metrics = trainer.evaluate(test_loader)
+45
View File
@@ -0,0 +1,45 @@
Traceback (most recent call last):
File "/Users/xxz/miniforge3/envs/crypto_pred_312/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 3641, in get_loc
return self._engine.get_loc(casted_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pandas/_libs/index.pyx", line 168, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 197, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 7668, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 7676, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'BBU_20_2.0'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/xxz/PycharmProjects/crypto_predictor/train_dlinear_real.py", line 163, in <module>
main()
File "/Users/xxz/PycharmProjects/crypto_predictor/train_dlinear_real.py", line 30, in main
X, y, processed_df = preprocessor.preprocess_pipeline(df, task='classification', fit=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/xxz/PycharmProjects/crypto_predictor/data/preprocessor.py", line 225, in preprocess_pipeline
df = self.add_technical_indicators(df)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/xxz/PycharmProjects/crypto_predictor/data/preprocessor.py", line 58, in add_technical_indicators
df['bb_upper'] = bbands['BBU_20_2.0']
~~~~~~^^^^^^^^^^^^^^
File "/Users/xxz/miniforge3/envs/crypto_pred_312/lib/python3.12/site-packages/pandas/core/frame.py", line 4378, in __getitem__
indexer = self.columns.get_loc(key)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/xxz/miniforge3/envs/crypto_pred_312/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 3648, in get_loc
raise KeyError(key) from err
KeyError: 'BBU_20_2.0'
ERROR conda.cli.main_run:execute(127): `conda run python train_dlinear_real.py` failed. (See above for error)
================================================================================
DLinear模型训练 - 真实Binance数据
================================================================================
[1/7] 从Binance获取BTC 1h真实数据...
✓ 使用SOCKS5代理: socks5://127.0.0.1:7890
✓ 已成功连接到Binance API
从缓存加载数据:data/cache/BTC_1h_1y_now.csv
✓ 已获取 8760 行真实数据
✓ 时间范围:2025-01-28 15:00:00 至 2026-01-28 14:00:00
[2/7] 数据预处理(技术指标 + 特征工程)...
+1349
View File
File diff suppressed because it is too large. Load diff