当前位置:网站首页>Auto Sharding Policy will apply Data Sharding policy as it failed to apply file Sharding Policy
Auto Sharding Policy will apply Data Sharding policy as it failed to apply file Sharding Policy
2022-06-26 15:36:00 【There2belief】
Utilisertf.distribute.MirroredStrategy()Heure,Avertissement:
AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset
Les codes sont les suivants::
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
keras_model = build_model()
train_datagen = ImageDataGenerator()
training_img_generator = train_datagen.flow_from_directory(
input_path,
target_size=(image_size, image_size),
batch_size=batch_size,
class_mode="categorical",
)
train_dataset = tf.data.Dataset.from_generator(
lambda: training_img_generator,
output_types=(tf.float32, tf.float32),
output_shapes=([None, image_size, image_size, 3], [None, len(image_classes)])
)
# similar for validation_dataset = ...
keras_model.fit(
train_dataset,
steps_per_epoch=train_steps_per_epoch,
epochs=epoch_count,
validation_data=validation_dataset,
validation_steps=validation_steps_per_epoch,
)Now this seem to work, the model is trained as usual. However, during training I get the following warning message, when using a mirrored strategy:
AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset
Tentative de résolution mais non valable
So I added the following lines between creating the data sets and calling fit():
options = tf.data.Options()
options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.DATA
train_dataset.with_options(options)
validation_dataset.with_options(options)However, I still get the same warning.
This leads me to these two questions:
- What do I need to do in order to get rid of this warning?
- Even more important: Why is TF not able to split the dataset with the default
AutoShardPolicy.FILEpolicy, since I am using thousands of images per class in the input folder?
边栏推荐
- Notes on brushing questions (19) -- binary tree: modification and construction of binary search tree
- [tcapulusdb knowledge base] tcapulusdb system user group introduction
- MongoDB系列之适用场景和不适用场景
- [wechat applet] event binding, do you understand?
- vsomeip3 双机通信文件配置
- 整理了一批脚本标准的函数模块(2021版)
- How to handle 2gcsv files that cannot be opened? Use byzer
- 通过券商经理的开户链接开股票账户安全吗?还是去证券公司开户安全?
- 学习内存屏障
- 在重新格式化时不要删除自定义换行符(Don‘t remove custom line breaks on reformat)
猜你喜欢

Particle filter PF -- Application in maneuvering target tracking (particle filter vs extended Kalman filter)

【ceph】CephFS 内部实现(二):示例--未消化

音视频学习(一)——PTZ控制原理

【ceph】mkdir|mksnap流程源码分析|锁状态切换实例

English语法_形容词/副词3级 - 原级句型

Use of abortcontroller
![[tcapulusdb knowledge base] tcapulusdb OMS business personnel permission introduction](/img/7b/8c4f1549054ee8c0184495d9e8e378.png)
[tcapulusdb knowledge base] tcapulusdb OMS business personnel permission introduction
![[CEPH] Lock Notes of cephfs](/img/9a/b68e7b07b1187794c0dbed36eea749.png)
[CEPH] Lock Notes of cephfs
Advanced operation of MySQL database basic SQL statement tutorial

Database - integrity constraints
随机推荐
Inaccurate data accuracy in ETL process
Ansible自动化的运用
IDEA本地代理后,无法下载插件
MongoDB系列之适用场景和不适用场景
Evaluate:huggingface detailed introduction to the evaluation index module
[CEPH] Introduction to cephfs caps
/etc/profile、/etc/bashrc、~/.bashrc的区别
【TcaplusDB知识库】TcaplusDB系统管理介绍
Unable to download Plug-in after idea local agent
[CEPH] MKDIR | mksnap process source code analysis | lock state switching example
JS simple deepcopy (Introduction recursion)
【TcaplusDB知识库】TcaplusDB单据受理-建表审批介绍
5张图诠释了容器网络
[tcapulusdb knowledge base] Introduction to tcapulusdb data structure
[wechat applet] event binding, do you understand?
Advanced operation of MySQL database basic SQL statement tutorial
[C language practice - printing hollow upper triangle and its deformation]
反射修改final
【leetcode】112. 路径总和 - 113. 路径总和 II
Smoothing data using convolution