Mentions légales du service
Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
S
Source_Encoding
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Deploy
Releases
Package registry
Model registry
Operate
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Admin message
GitLab upgrade completed. Current version is 17.11.3.
Show more breadcrumbs
dnarXiv
Source_Encoding
Commits
acf319ec
Commit
acf319ec
authored
1 year ago
by
BOULLE Olivier
Browse files
Options
Downloads
Patches
Plain Diff
fix case when file has '__' in base name for real this time
parent
2021d122
No related branches found
No related tags found
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
post_processing.py
+15
-12
15 additions, 12 deletions
post_processing.py
with
15 additions
and
12 deletions
post_processing.py
+
15
−
12
View file @
acf319ec
...
...
@@ -102,18 +102,21 @@ def replace_files(uncompressed_dir_path, original_files_dir_path):
# directories already exists
pass
if
"
__
"
in
header_line
and
"
/
"
in
header_line
:
# means the file is a split file that needs to be merged with the corresponding other split files
subfile_number_str
,
subfile_total_str
=
header_line
.
split
(
"
__
"
)[
1
].
split
(
"
/
"
)
# get the number of splits and the split index for this subfile
subfile_number
,
subfile_total
=
int
(
subfile_number_str
),
int
(
subfile_total_str
)
# warning, subfile count starts from 1
# get the list of subfiles lines for the same file, or init the list with size = number of subfiles
split_files_dict
[
original_file_path
]
=
split_files_dict
.
get
(
original_file_path
,
[
""
for
i
in
range
(
subfile_total
)])
if
split_files_dict
[
original_file_path
][
subfile_number
-
1
]
!=
""
:
# use subfile_number-1 because it starts from 1 and not 0
print
(
"
warning post processing (replace_files) : a subfile with the same number already exists
"
,
filename
,
original_file_path
,
subfile_number
)
else
:
# save the bytes of this subfile
split_files_dict
[
original_file_path
][
subfile_number
-
1
]
=
b
""
.
join
(
small_file_bytes_lines
[
1
:])
if
"
__
"
in
header_line
:
# means the file is a split file that needs to be merged with the corresponding other split files
# should be like "path/basename__1/3"
header_counter
=
header_line
.
split
(
"
__
"
)[
-
1
]
if
"
/
"
in
header_counter
:
# if the '__' was in the basename of the file, it's not for the counter
subfile_number_str
,
subfile_total_str
=
header_counter
.
split
(
"
/
"
)
# get the number of splits and the split index for this subfile
subfile_number
,
subfile_total
=
int
(
subfile_number_str
),
int
(
subfile_total_str
)
# warning, subfile count starts from 1
# get the list of subfiles lines for the same file, or init the list with size = number of subfiles
split_files_dict
[
original_file_path
]
=
split_files_dict
.
get
(
original_file_path
,
[
""
for
i
in
range
(
subfile_total
)])
if
split_files_dict
[
original_file_path
][
subfile_number
-
1
]
!=
""
:
# use subfile_number-1 because it starts from 1 and not 0
print
(
"
warning post processing (replace_files) : a subfile with the same number already exists
"
,
filename
,
original_file_path
,
subfile_number
)
else
:
# save the bytes of this subfile
split_files_dict
[
original_file_path
][
subfile_number
-
1
]
=
b
""
.
join
(
small_file_bytes_lines
[
1
:])
else
:
# write the content without the header line in the file
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment