devbox
凡人多烦事01
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

erofs 阅读_cannot find valid erofs superblock

作者：凡人多烦事01 | 2024-03-25 10:27:23

赞

踩

cannot find valid erofs superblock

Overview
========

EROFS file-system stands for Enhanced Read-Only File System. Different
from other read-only file systems, it aims to be designed for flexibility,
scalability, but be kept simple and high performance.

It is designed as a better filesystem solution for the following scenarios:

 - read-only storage media or

 - part of a fully trusted read-only solution, which means it needs to be
   immutable and bit-for-bit identical to the official golden image for
   their releases due to security and other considerations and

 - hope to save some extra storage space with guaranteed end-to-end performance
   by using reduced metadata and transparent file compression, especially
   for those embedded devices with limited memory (ex, smartphone);

Here is the main features of EROFS:

 - Little endian on-disk design;

 - Currently 4KB block size (nobh) and therefore maximum 16TB address space;

 - Metadata & data could be mixed by design;

 - 2 inode versions for different requirements:

   =====================  ============  =====================================
                          compact (v1)  extended (v2)
   =====================  ============  =====================================
   Inode metadata size    32 bytes      64 bytes
   Max file size          4 GB          16 EB (also limited by max. vol size)
   Max uids/gids          65536         4294967296
   File change time       no            yes (64 + 32-bit timestamp)
   Max hardlinks          65536         4294967296
   Metadata reserved      4 bytes       14 bytes
   =====================  ============  =====================================

 - Support extended attributes (xattrs) as an option;

 - Support xattr inline and tail-end data inline for all files;

 - Support POSIX.1e ACLs by using xattrs;

 - Support transparent data compression as an option:
   LZ4 algorithm with the fixed-sized output compression for high performance.

The following git tree provides the file system user-space tools under
development (ex, formatting tool mkfs.erofs):

- git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git

Bugs and patches are welcome, please kindly help us and send to the following
linux-erofs mailing list:

- linux-erofs mailing list   <linux-erofs@lists.ozlabs.org>

Mount options
=============

===================    =========================================================
(no)user_xattr         Setup Extended User Attributes. Note: xattr is enabled
                       by default if CONFIG_EROFS_FS_XATTR is selected.
(no)acl                Setup POSIX Access Control List. Note: acl is enabled
                       by default if CONFIG_EROFS_FS_POSIX_ACL is selected.
cache_strategy=%s      Select a strategy for cached decompression from now on:

               ==========  =============================================
                         disabled  In-place I/O decompression only;
                        readahead  Cache the last incomplete compressed physical
                                   cluster for further reading. It still does
                                   in-place I/O decompression for the rest
                                   compressed physical clusters;
                       readaround  Cache the both ends of incomplete compressed
                                   physical clusters for further reading.
                                   It still does in-place I/O decompression
                                   for the rest compressed physical clusters.
               ==========  =============================================
dax={always,never}     Use direct access (no page cache).  See
                       Documentation/filesystems/dax.rst.
dax                    A legacy option which is an alias for ``dax=always``.
===================    =========================================================

On-disk details
===============

Summary
-------
Different from other read-only file systems, an EROFS volume is designed
to be as simple as possible::

                                |-> aligned with the block size
   ____________________________________________________________
  | |SB| | ... | Metadata | ... | Data | Metadata | ... | Data |
  |_|__|_|_____|__________|_____|______|__________|_____|______|
  0 +1K

All data areas should be aligned with the block size, but metadata areas
may not. All metadatas can be now observed in two different spaces (views):

 1. Inode metadata space

    Each valid inode should be aligned with an inode slot, which is a fixed
    value (32 bytes) and designed to be kept in line with compact inode size.

    Each inode can be directly found with the following formula:
         inode offset = meta_blkaddr * block_size + 32 * nid

    ::

                                 |-> aligned with 8B
                                            |-> followed closely
     + meta_blkaddr blocks                                      |-> another slot
       _____________________________________________________________________
     |  ...   | inode |  xattrs  | extents  | data inline | ... | inode ...
     |________|_______|(optional)|(optional)|__(optional)_|_____|__________
              |-> aligned with the inode slot size
                   .                   .
                 .                         .
               .                              .
             .                                    .
           .                                         .
         .                                              .
       .____________________________________________________|-> aligned with 4B
       | xattr_ibody_header | shared xattrs | inline xattrs |
       |____________________|_______________|_______________|
       |->    12 bytes    <-|->x * 4 bytes<-|               .
                           .                .                 .
                     .                      .                   .
                .                           .                     .
            ._______________________________.______________________.
            | id | id | id | id |  ... | id | ent | ... | ent| ... |
            |____|____|____|____|______|____|_____|_____|____|_____|
                                            |-> aligned with 4B
                                                        |-> aligned with 4B

    Inode could be 32 or 64 bytes, which can be distinguished from a common
    field which all inode versions have -- i_format::

        __________________               __________________
       |     i_format     |             |     i_format     |
       |__________________|             |__________________|
       |        ...       |             |        ...       |
       |                  |             |                  |
       |__________________| 32 bytes    |                  |
                                        |                  |
                                        |__________________| 64 bytes

    Xattrs, extents, data inline are followed by the corresponding inode with
    proper alignment, and they could be optional for different data mappings.
    _currently_ total 5 data layouts are supported:

    ==  ====================================================================
     0  flat file data without data inline (no extent);
     1  fixed-sized output data compression (with non-compacted indexes);
     2  flat file data with tail packing data inline (no extent);
     3  fixed-sized output data compression (with compacted indexes, v5.3+);
     4  chunk-based file (v5.15+).
    ==  ====================================================================

    The size of the optional xattrs is indicated by i_xattr_count in inode
    header. Large xattrs or xattrs shared by many different files can be
    stored in shared xattrs metadata rather than inlined right after inode.

 2. Shared xattrs metadata space

    Shared xattrs space is similar to the above inode space, started with
    a specific block indicated by xattr_blkaddr, organized one by one with
    proper align.

    Each share xattr can also be directly found by the following formula:
         xattr offset = xattr_blkaddr * block_size + 4 * xattr_id

::

                           |-> aligned by  4 bytes
    + xattr_blkaddr blocks                     |-> aligned with 4 bytes
     _________________________________________________________________________
    |  ...   | xattr_entry |  xattr data | ... |  xattr_entry | xattr data  ...
    |________|_____________|_____________|_____|______________|_______________

Directories
-----------
All directories are now organized in a compact on-disk format. Note that
each directory block is divided into index and name areas in order to support
random file lookup, and all directory entries are _strictly_ recorded in
alphabetical order in order to support improved prefix binary search
algorithm (could refer to the related source code).

::

                  ___________________________
                 /                           |
                /              ______________|________________
               /              /              | nameoff1       | nameoffN-1
  ____________.______________._______________v________________v__________
 | dirent | dirent | ... | dirent | filename | filename | ... | filename |
 |___.0___|____1___|_____|___N-1__|____0_____|____1_____|_____|___N-1____|
      \                           ^
       \                          |                           * could have
        \                         |                             trailing '\0'
         \________________________| nameoff0
                             Directory block

Note that apart from the offset of the first filename, nameoff0 also indicates
the total number of directory entries in this block since it is no need to
introduce another on-disk field at all.

Chunk-based file
----------------
In order to support chunk-based data deduplication, a new inode data layout has
been supported since Linux v5.15: Files are split in equal-sized data chunks
with ``extents`` area of the inode metadata indicating how to get the chunk
data: these can be simply as a 4-byte block address array or in the 8-byte
chunk index form (see struct erofs_inode_chunk_index in erofs_fs.h for more
details.)

By the way, chunk-based files are all uncompressed for now.

Data compression
----------------
EROFS implements LZ4 fixed-sized output compression which generates fixed-sized
compressed data blocks from variable-sized input in contrast to other existing
fixed-sized input solutions. Relatively higher compression ratios can be gotten
by using fixed-sized output compression since nowadays popular data compression
algorithms are mostly LZ77-based and such fixed-sized output approach can be
benefited from the historical dictionary (aka. sliding window).

In details, original (uncompressed) data is turned into several variable-sized
extents and in the meanwhile, compressed into physical clusters (pclusters).
In order to record each variable-sized extent, logical clusters (lclusters) are
introduced as the basic unit of compress indexes to indicate whether a new
extent is generated within the range (HEAD) or not (NONHEAD). Lclusters are now
fixed in block size, as illustrated below::

          |<-    variable-sized extent    ->|<-       VLE         ->|
        clusterofs                        clusterofs              clusterofs
          |                                 |                       |
 _________v_________________________________v_______________________v________
 ... |    .         |              |        .     |              |  .   ...
 ____|____._________|______________|________.___ _|______________|__.________
     |-> lcluster <-|-> lcluster <-|-> lcluster <-|-> lcluster <-|
          (HEAD)        (NONHEAD)       (HEAD)        (NONHEAD)    .
           .             CBLKCNT            .                    .
            .                               .                  .
             .                              .                .
       _______._____________________________.______________._________________
          ... |              |              |              | ...
       _______|______________|______________|______________|_________________
              |->      big pcluster       <-|-> pcluster <-|

A physical cluster can be seen as a container of physical compressed blocks
which contains compressed data. Previously, only lcluster-sized (4KB) pclusters
were supported. After big pcluster feature is introduced (available since
Linux v5.13), pcluster can be a multiple of lcluster size.

For each HEAD lcluster, clusterofs is recorded to indicate where a new extent
starts and blkaddr is used to seek the compressed data. For each NONHEAD
lcluster, delta0 and delta1 are available instead of blkaddr to indicate the
distance to its HEAD lcluster and the next HEAD lcluster. A PLAIN lcluster is
also a HEAD lcluster except that its data is uncompressed. See the comments
around "struct z_erofs_vle_decompressed_index" in erofs_fs.h for more details.

If big pcluster is enabled, pcluster size in lclusters needs to be recorded as
well. Let the delta0 of the first NONHEAD lcluster store the compressed block
count with a special flag as a new called CBLKCNT NONHEAD lcluster. It's easy
to understand its delta0 is constantly 1, as illustrated below::

   __________________________________________________________
  | HEAD |  NONHEAD  | NONHEAD | ... | NONHEAD | HEAD | HEAD |
  |__:___|_(CBLKCNT)_|_________|_____|_________|__:___|____:_|
     |<----- a big pcluster (with CBLKCNT) ------>|<--  -->|
           a lcluster-sized pcluster (without CBLKCNT) ^

If another HEAD follows a HEAD lcluster, there is no room to record CBLKCNT,
but it's easy to know the size of such pcluster is 1 lcluster as well.

drivers\staging\erofs\erofs_fs.h

  1. /* SPDX-License-Identifier: GPL-2.0 OR Apache-2.0
  2.  *
  3.  * linux/drivers/staging/erofs/erofs_fs.h
  4.  *
  5.  * Copyright (C) 2017-2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is dual-licensed; you may select either the GNU General Public
  10.  * License version 2 or Apache License, Version 2.0. See the file COPYING
  11.  * in the main directory of the Linux distribution for more details.
  12.  */
  13. #ifndef __EROFS_FS_H
  14. #define __EROFS_FS_H
  15.  
  16. /* Enhanced(Extended) ROM File System */
  17. #define EROFS_SUPER_MAGIC_V1    0xE0F5E1E2
  18. #define EROFS_SUPER_OFFSET      1024
  19.  
  20. /*
  21.  * Any bits that aren't in EROFS_ALL_REQUIREMENTS should be
  22.  * incompatible with this kernel version.
  23.  */
  24. #define EROFS_ALL_REQUIREMENTS  0
  25.  
  26. struct erofs_super_block {
  27. /*  0 */__le32 magic;           /* in the little endian */
  28. /*  4 */__le32 checksum;        /* crc32c(super_block) */
  29. /*  8 */__le32 features;        /* (aka. feature_compat) */
  30. /* 12 */__u8 blkszbits;         /* support block_size == PAGE_SIZE only */
  31. /* 13 */__u8 reserved;
  32.  
  33. /* 14 */__le16 root_nid;
  34. /* 16 */__le64 inos;            /* total valid ino # (== f_files - f_favail) */
  35.  
  36. /* 24 */__le64 build_time;      /* inode v1 time derivation */
  37. /* 32 */__le32 build_time_nsec;
  38. /* 36 */__le32 blocks;          /* used for statfs */
  39. /* 40 */__le32 meta_blkaddr;
  40. /* 44 */__le32 xattr_blkaddr;
  41. /* 48 */__u8 uuid[16];          /* 128-bit uuid for volume */
  42. /* 64 */__u8 volume_name[16];   /* volume name */
  43. /* 80 */__le32 requirements;    /* (aka. feature_incompat) */
  44.  
  45. /* 84 */__u8 reserved2[44];
  46. } __packed;                     /* 128 bytes */
  47.  
  48. #define __EROFS_BIT(_prefix, _cur, _pre)	enum {	\
  49. 	_prefix ## _cur ## _BIT = _prefix ## _pre ## _BIT + \
  50. 		_prefix ## _pre ## _BITS }
  51.  
  52. /*
  53.  * erofs inode data mapping:
  54.  * 0 - inode plain without inline data A:
  55.  * inode, [xattrs], ... | ... | no-holed data
  56.  * 1 - inode VLE compression B:
  57.  * inode, [xattrs], extents ... | ...
  58.  * 2 - inode plain with inline data C:
  59.  * inode, [xattrs], last_inline_data, ... | ... | no-holed data
  60.  * 3~7 - reserved
  61.  */
  62. enum {
  63. 	EROFS_INODE_LAYOUT_PLAIN,
  64. 	EROFS_INODE_LAYOUT_COMPRESSION,
  65. 	EROFS_INODE_LAYOUT_INLINE,
  66. 	EROFS_INODE_LAYOUT_MAX
  67. };
  68. #define EROFS_I_VERSION_BITS            1
  69. #define EROFS_I_DATA_MAPPING_BITS       3
  70.  
  71. #define EROFS_I_VERSION_BIT             0
  72. __EROFS_BIT(EROFS_I_, DATA_MAPPING, VERSION);
  73.  
  74. #define EROFS_I_ALL	\
  75. 	((1 << (EROFS_I_DATA_MAPPING_BIT + EROFS_I_DATA_MAPPING_BITS)) - 1)
  76.  
  77. struct erofs_inode_v1 {
  78. /*  0 */__le16 i_advise;
  79.  
  80. /* 1 header + n-1 * 4 bytes inline xattr to keep continuity */
  81. /*  2 */__le16 i_xattr_icount;
  82. /*  4 */__le16 i_mode;
  83. /*  6 */__le16 i_nlink;
  84. /*  8 */__le32 i_size;
  85. /* 12 */__le32 i_reserved;
  86. /* 16 */union {
  87. 		/* file total compressed blocks for data mapping 1 */
  88. 		__le32 compressed_blocks;
  89. 		__le32 raw_blkaddr;
  90.  
  91. 		/* for device files, used to indicate old/new device # */
  92. 		__le32 rdev;
  93. 	} i_u __packed;
  94. /* 20 */__le32 i_ino;           /* only used for 32-bit stat compatibility */
  95. /* 24 */__le16 i_uid;
  96. /* 26 */__le16 i_gid;
  97. /* 28 */__le32 i_checksum;
  98. } __packed;
  99.  
  100. /* 32 bytes on-disk inode */
  101. #define EROFS_INODE_LAYOUT_V1   0
  102. /* 64 bytes on-disk inode */
  103. #define EROFS_INODE_LAYOUT_V2   1
  104.  
  105. struct erofs_inode_v2 {
  106. 	__le16 i_advise;
  107.  
  108. 	/* 1 header + n-1 * 4 bytes inline xattr to keep continuity */
  109. 	__le16 i_xattr_icount;
  110. 	__le16 i_mode;
  111. 	__le16 i_reserved;      /* 8 bytes */
  112. 	__le64 i_size;          /* 16 bytes */
  113. 	union {
  114. 		/* file total compressed blocks for data mapping 1 */
  115. 		__le32 compressed_blocks;
  116. 		__le32 raw_blkaddr;
  117.  
  118. 		/* for device files, used to indicate old/new device # */
  119. 		__le32 rdev;
  120. 	} i_u __packed;
  121.  
  122. 	/* only used for 32-bit stat compatibility */
  123. 	__le32 i_ino;           /* 24 bytes */
  124.  
  125. 	__le32 i_uid;
  126. 	__le32 i_gid;
  127. 	__le64 i_ctime;         /* 32 bytes */
  128. 	__le32 i_ctime_nsec;
  129. 	__le32 i_nlink;
  130. 	__u8   i_reserved2[12];
  131. 	__le32 i_checksum;      /* 64 bytes */
  132. } __packed;
  133.  
  134. #define EROFS_MAX_SHARED_XATTRS         (128)
  135. /* h_shared_count between 129 ... 255 are special # */
  136. #define EROFS_SHARED_XATTR_EXTENT       (255)
  137.  
  138. /*
  139.  * inline xattrs (n == i_xattr_icount):
  140.  * erofs_xattr_ibody_header(1) + (n - 1) * 4 bytes
  141.  *          12 bytes           /                   \
  142.  *                            /                     \
  143.  *                           /-----------------------\
  144.  *                           |  erofs_xattr_entries+ |
  145.  *                           +-----------------------+
  146.  * inline xattrs must starts in erofs_xattr_ibody_header,
  147.  * for read-only fs, no need to introduce h_refcount
  148.  */
  149. struct erofs_xattr_ibody_header {
  150. 	__le32 h_checksum;
  151. 	__u8   h_shared_count;
  152. 	__u8   h_reserved[7];
  153. 	__le32 h_shared_xattrs[0];      /* shared xattr id array */
  154. } __packed;
  155.  
  156. /* Name indexes */
  157. #define EROFS_XATTR_INDEX_USER              1
  158. #define EROFS_XATTR_INDEX_POSIX_ACL_ACCESS  2
  159. #define EROFS_XATTR_INDEX_POSIX_ACL_DEFAULT 3
  160. #define EROFS_XATTR_INDEX_TRUSTED           4
  161. #define EROFS_XATTR_INDEX_LUSTRE            5
  162. #define EROFS_XATTR_INDEX_SECURITY          6
  163.  
  164. /* xattr entry (for both inline & shared xattrs) */
  165. struct erofs_xattr_entry {
  166. 	__u8   e_name_len;      /* length of name */
  167. 	__u8   e_name_index;    /* attribute name index */
  168. 	__le16 e_value_size;    /* size of attribute value */
  169. 	/* followed by e_name and e_value */
  170. 	char   e_name[0];       /* attribute name */
  171. } __packed;
  172.  
  173. #define ondisk_xattr_ibody_size(count)	({\
  174. 	u32 __count = le16_to_cpu(count); \
  175. 	((__count) == 0) ? 0 : \
  176. 	sizeof(struct erofs_xattr_ibody_header) + \
  177. 		sizeof(__u32) * ((__count) - 1); })
  178.  
  179. #define EROFS_XATTR_ALIGN(size) round_up(size, sizeof(struct erofs_xattr_entry))
  180. #define EROFS_XATTR_ENTRY_SIZE(entry) EROFS_XATTR_ALIGN( \
  181. 	sizeof(struct erofs_xattr_entry) + \
  182. 	(entry)->e_name_len + le16_to_cpu((entry)->e_value_size))
  183.  
  184. /* have to be aligned with 8 bytes on disk */
  185. struct erofs_extent_header {
  186. 	__le32 eh_checksum;
  187. 	__le32 eh_reserved[3];
  188. } __packed;
  189.  
  190. /*
  191.  * Z_EROFS Variable-sized Logical Extent cluster type:
  192.  *    0 - literal (uncompressed) cluster
  193.  *    1 - compressed cluster (for the head logical cluster)
  194.  *    2 - compressed cluster (for the other logical clusters)
  195.  *
  196.  * In detail,
  197.  *    0 - literal (uncompressed) cluster,
  198.  *        di_advise = 0
  199.  *        di_clusterofs = the literal data offset of the cluster
  200.  *        di_blkaddr = the blkaddr of the literal cluster
  201.  *
  202.  *    1 - compressed cluster (for the head logical cluster)
  203.  *        di_advise = 1
  204.  *        di_clusterofs = the decompressed data offset of the cluster
  205.  *        di_blkaddr = the blkaddr of the compressed cluster
  206.  *
  207.  *    2 - compressed cluster (for the other logical clusters)
  208.  *        di_advise = 2
  209.  *        di_clusterofs =
  210.  *           the decompressed data offset in its own head cluster
  211.  *        di_u.delta[0] = distance to its corresponding head cluster
  212.  *        di_u.delta[1] = distance to its corresponding tail cluster
  213.  *                (di_advise could be 0, 1 or 2)
  214.  */
  215. #define Z_EROFS_VLE_DI_CLUSTER_TYPE_BITS        2
  216. #define Z_EROFS_VLE_DI_CLUSTER_TYPE_BIT         0
  217.  
  218. struct z_erofs_vle_decompressed_index {
  219. 	__le16 di_advise;
  220. 	/* where to decompress in the head cluster */
  221. 	__le16 di_clusterofs;
  222.  
  223. 	union {
  224. 		/* for the head cluster */
  225. 		__le32 blkaddr;
  226. 		/*
  227. 		 * for the rest clusters
  228. 		 * eg. for 4k page-sized cluster, maximum 4K*64k = 256M)
  229. 		 * [0] - pointing to the head cluster
  230. 		 * [1] - pointing to the tail cluster
  231. 		 */
  232. 		__le16 delta[2];
  233. 	} di_u __packed;		/* 8 bytes */
  234. } __packed;
  235.  
  236. #define Z_EROFS_VLE_EXTENT_ALIGN(size) round_up(size, \
  237. 	sizeof(struct z_erofs_vle_decompressed_index))
  238.  
  239. /* dirent sorts in alphabet order, thus we can do binary search */
  240. struct erofs_dirent {
  241. 	__le64 nid;     /*  0, node number */
  242. 	__le16 nameoff; /*  8, start offset of file name */
  243. 	__u8 file_type; /* 10, file type */
  244. 	__u8 reserved;  /* 11, reserved */
  245. } __packed;
  246.  
  247. /* file types used in inode_info->flags */
  248. enum {
  249. 	EROFS_FT_UNKNOWN,
  250. 	EROFS_FT_REG_FILE,
  251. 	EROFS_FT_DIR,
  252. 	EROFS_FT_CHRDEV,
  253. 	EROFS_FT_BLKDEV,
  254. 	EROFS_FT_FIFO,
  255. 	EROFS_FT_SOCK,
  256. 	EROFS_FT_SYMLINK,
  257. 	EROFS_FT_MAX
  258. };
  259.  
  260. #define EROFS_NAME_LEN      255
  261.  
  262. /* check the EROFS on-disk layout strictly at compile time */
  263. static inline void erofs_check_ondisk_layout_definitions(void)
  264. {
  265. 	BUILD_BUG_ON(sizeof(struct erofs_super_block) != 128);
  266. 	BUILD_BUG_ON(sizeof(struct erofs_inode_v1) != 32);
  267. 	BUILD_BUG_ON(sizeof(struct erofs_inode_v2) != 64);
  268. 	BUILD_BUG_ON(sizeof(struct erofs_xattr_ibody_header) != 12);
  269. 	BUILD_BUG_ON(sizeof(struct erofs_xattr_entry) != 4);
  270. 	BUILD_BUG_ON(sizeof(struct erofs_extent_header) != 16);
  271. 	BUILD_BUG_ON(sizeof(struct z_erofs_vle_decompressed_index) != 8);
  272. 	BUILD_BUG_ON(sizeof(struct erofs_dirent) != 12);
  273. }
  274.  
  275. #endif
  276.

internal.h

  1. /* SPDX-License-Identifier: GPL-2.0
  2.  *
  3.  * linux/drivers/staging/erofs/internal.h
  4.  *
  5.  * Copyright (C) 2017-2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #ifndef __INTERNAL_H
  14. #define __INTERNAL_H
  15.  
  16. #include <linux/fs.h>
  17. #include <linux/dcache.h>
  18. #include <linux/mm.h>
  19. #include <linux/pagemap.h>
  20. #include <linux/bio.h>
  21. #include <linux/buffer_head.h>
  22. #include <linux/cleancache.h>
  23. #include <linux/slab.h>
  24. #include <linux/vmalloc.h>
  25. #include "erofs_fs.h"
  26.  
  27. /* redefine pr_fmt "erofs: " */
  28. #undef pr_fmt
  29. #define pr_fmt(fmt) "erofs: " fmt
  30.  
  31. #define errln(x, ...)   pr_err(x "\n", ##__VA_ARGS__)
  32. #define infoln(x, ...)  pr_info(x "\n", ##__VA_ARGS__)
  33. #ifdef CONFIG_EROFS_FS_DEBUG
  34. #define debugln(x, ...) pr_debug(x "\n", ##__VA_ARGS__)
  35.  
  36. #define dbg_might_sleep         might_sleep
  37. #define DBG_BUGON               BUG_ON
  38. #else
  39. #define debugln(x, ...)         ((void)0)
  40.  
  41. #define dbg_might_sleep()       ((void)0)
  42. #define DBG_BUGON(x)            ((void)(x))
  43. #endif
  44.  
  45. #ifdef CONFIG_EROFS_FAULT_INJECTION
  46. enum {
  47. 	FAULT_KMALLOC,
  48. 	FAULT_MAX,
  49. };
  50.  
  51. extern char *erofs_fault_name[FAULT_MAX];
  52. #define IS_FAULT_SET(fi, type) ((fi)->inject_type & (1 << (type)))
  53.  
  54. struct erofs_fault_info {
  55. 	atomic_t inject_ops;
  56. 	unsigned int inject_rate;
  57. 	unsigned int inject_type;
  58. };
  59. #endif
  60.  
  61. #ifdef CONFIG_EROFS_FS_ZIP_CACHE_BIPOLAR
  62. #define EROFS_FS_ZIP_CACHE_LVL	(2)
  63. #elif defined(EROFS_FS_ZIP_CACHE_UNIPOLAR)
  64. #define EROFS_FS_ZIP_CACHE_LVL	(1)
  65. #else
  66. #define EROFS_FS_ZIP_CACHE_LVL	(0)
  67. #endif
  68.  
  69. #if (!defined(EROFS_FS_HAS_MANAGED_CACHE) && (EROFS_FS_ZIP_CACHE_LVL > 0))
  70. #define EROFS_FS_HAS_MANAGED_CACHE
  71. #endif
  72.  
  73. /* EROFS_SUPER_MAGIC_V1 to represent the whole file system */
  74. #define EROFS_SUPER_MAGIC   EROFS_SUPER_MAGIC_V1
  75.  
  76. typedef u64 erofs_nid_t;
  77.  
  78. struct erofs_sb_info {
  79. 	/* list for all registered superblocks, mainly for shrinker */
  80. 	struct list_head list;
  81. 	struct mutex umount_mutex;
  82.  
  83. 	u32 blocks;
  84. 	u32 meta_blkaddr;
  85. #ifdef CONFIG_EROFS_FS_XATTR
  86. 	u32 xattr_blkaddr;
  87. #endif
  88.  
  89. 	/* inode slot unit size in bit shift */
  90. 	unsigned char islotbits;
  91. #ifdef CONFIG_EROFS_FS_ZIP
  92. 	/* cluster size in bit shift */
  93. 	unsigned char clusterbits;
  94.  
  95. 	/* the dedicated workstation for compression */
  96. 	struct radix_tree_root workstn_tree;
  97.  
  98. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  99. 	struct inode *managed_cache;
  100. #endif
  101.  
  102. #endif
  103.  
  104. 	u32 build_time_nsec;
  105. 	u64 build_time;
  106.  
  107. 	/* what we really care is nid, rather than ino.. */
  108. 	erofs_nid_t root_nid;
  109. 	/* used for statfs, f_files - f_favail */
  110. 	u64 inos;
  111.  
  112. 	u8 uuid[16];                    /* 128-bit uuid for volume */
  113. 	u8 volume_name[16];             /* volume name */
  114. 	u32 requirements;
  115.  
  116. 	char *dev_name;
  117.  
  118. 	unsigned int mount_opt;
  119. 	unsigned int shrinker_run_no;
  120.  
  121. #ifdef CONFIG_EROFS_FAULT_INJECTION
  122. 	struct erofs_fault_info fault_info;	/* For fault injection */
  123. #endif
  124. };
  125.  
  126. #ifdef CONFIG_EROFS_FAULT_INJECTION
  127. #define erofs_show_injection_info(type)					\
  128. 	infoln("inject %s in %s of %pS", erofs_fault_name[type],        \
  129. 		__func__, __builtin_return_address(0))
  130.  
  131. static inline bool time_to_inject(struct erofs_sb_info *sbi, int type)
  132. {
  133. 	struct erofs_fault_info *ffi = &sbi->fault_info;
  134.  
  135. 	if (!ffi->inject_rate)
  136. 		return false;
  137.  
  138. 	if (!IS_FAULT_SET(ffi, type))
  139. 		return false;
  140.  
  141. 	atomic_inc(&ffi->inject_ops);
  142. 	if (atomic_read(&ffi->inject_ops) >= ffi->inject_rate) {
  143. 		atomic_set(&ffi->inject_ops, 0);
  144. 		return true;
  145. 	}
  146. 	return false;
  147. }
  148. #endif
  149.  
  150. static inline void *erofs_kmalloc(struct erofs_sb_info *sbi,
  151. 					size_t size, gfp_t flags)
  152. {
  153. #ifdef CONFIG_EROFS_FAULT_INJECTION
  154. 	if (time_to_inject(sbi, FAULT_KMALLOC)) {
  155. 		erofs_show_injection_info(FAULT_KMALLOC);
  156. 		return NULL;
  157. 	}
  158. #endif
  159. 	return kmalloc(size, flags);
  160. }
  161.  
  162. #define EROFS_SB(sb) ((struct erofs_sb_info *)(sb)->s_fs_info)
  163. #define EROFS_I_SB(inode) ((struct erofs_sb_info *)(inode)->i_sb->s_fs_info)
  164.  
  165. /* Mount flags set via mount options or defaults */
  166. #define EROFS_MOUNT_XATTR_USER		0x00000010
  167. #define EROFS_MOUNT_POSIX_ACL		0x00000020
  168. #define EROFS_MOUNT_FAULT_INJECTION	0x00000040
  169.  
  170. #define clear_opt(sbi, option)	((sbi)->mount_opt &= ~EROFS_MOUNT_##option)
  171. #define set_opt(sbi, option)	((sbi)->mount_opt |= EROFS_MOUNT_##option)
  172. #define test_opt(sbi, option)	((sbi)->mount_opt & EROFS_MOUNT_##option)
  173.  
  174. #ifdef CONFIG_EROFS_FS_ZIP
  175. #define erofs_workstn_lock(sbi)         xa_lock(&(sbi)->workstn_tree)
  176. #define erofs_workstn_unlock(sbi)       xa_unlock(&(sbi)->workstn_tree)
  177.  
  178. /* basic unit of the workstation of a super_block */
  179. struct erofs_workgroup {
  180. 	/* the workgroup index in the workstation */
  181. 	pgoff_t index;
  182.  
  183. 	/* overall workgroup reference count */
  184. 	atomic_t refcount;
  185. };
  186.  
  187. #define EROFS_LOCKED_MAGIC     (INT_MIN | 0xE0F510CCL)
  188.  
  189. #if defined(CONFIG_SMP)
  190. static inline bool erofs_workgroup_try_to_freeze(struct erofs_workgroup *grp,
  191. 						 int val)
  192. {
  193. 	preempt_disable();
  194. 	if (val != atomic_cmpxchg(&grp->refcount, val, EROFS_LOCKED_MAGIC)) {
  195. 		preempt_enable();
  196. 		return false;
  197. 	}
  198. 	return true;
  199. }
  200.  
  201. static inline void erofs_workgroup_unfreeze(struct erofs_workgroup *grp,
  202. 					    int orig_val)
  203. {
  204. 	/*
  205. 	 * other observers should notice all modifications
  206. 	 * in the freezing period.
  207. 	 */
  208. 	smp_mb();
  209. 	atomic_set(&grp->refcount, orig_val);
  210. 	preempt_enable();
  211. }
  212.  
  213. static inline int erofs_wait_on_workgroup_freezed(struct erofs_workgroup *grp)
  214. {
  215. 	return atomic_cond_read_relaxed(&grp->refcount,
  216. 					VAL != EROFS_LOCKED_MAGIC);
  217. }
  218. #else
  219. static inline bool erofs_workgroup_try_to_freeze(struct erofs_workgroup *grp,
  220. 						 int val)
  221. {
  222. 	preempt_disable();
  223. 	/* no need to spin on UP platforms, let's just disable preemption. */
  224. 	if (val != atomic_read(&grp->refcount)) {
  225. 		preempt_enable();
  226. 		return false;
  227. 	}
  228. 	return true;
  229. }
  230.  
  231. static inline void erofs_workgroup_unfreeze(struct erofs_workgroup *grp,
  232. 					    int orig_val)
  233. {
  234. 	preempt_enable();
  235. }
  236.  
  237. static inline int erofs_wait_on_workgroup_freezed(struct erofs_workgroup *grp)
  238. {
  239. 	int v = atomic_read(&grp->refcount);
  240.  
  241. 	/* workgroup is never freezed on uniprocessor systems */
  242. 	DBG_BUGON(v == EROFS_LOCKED_MAGIC);
  243. 	return v;
  244. }
  245. #endif
  246.  
  247. static inline bool erofs_workgroup_get(struct erofs_workgroup *grp, int *ocnt)
  248. {
  249. 	int o;
  250.  
  251. repeat:
  252. 	o = erofs_wait_on_workgroup_freezed(grp);
  253.  
  254. 	if (unlikely(o <= 0))
  255. 		return -1;
  256.  
  257. 	if (unlikely(atomic_cmpxchg(&grp->refcount, o, o + 1) != o))
  258. 		goto repeat;
  259.  
  260. 	*ocnt = o;
  261. 	return 0;
  262. }
  263.  
  264. #define __erofs_workgroup_get(grp)	atomic_inc(&(grp)->refcount)
  265. #define __erofs_workgroup_put(grp)	atomic_dec(&(grp)->refcount)
  266.  
  267. extern int erofs_workgroup_put(struct erofs_workgroup *grp);
  268.  
  269. extern struct erofs_workgroup *erofs_find_workgroup(
  270. 	struct super_block *sb, pgoff_t index, bool *tag);
  271.  
  272. extern int erofs_register_workgroup(struct super_block *sb,
  273. 	struct erofs_workgroup *grp, bool tag);
  274.  
  275. extern unsigned long erofs_shrink_workstation(struct erofs_sb_info *sbi,
  276. 	unsigned long nr_shrink, bool cleanup);
  277.  
  278. static inline void erofs_workstation_cleanup_all(struct super_block *sb)
  279. {
  280. 	erofs_shrink_workstation(EROFS_SB(sb), ~0UL, true);
  281. }
  282.  
  283. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  284. #define EROFS_UNALLOCATED_CACHED_PAGE	((void *)0x5F0EF00D)
  285.  
  286. extern int erofs_try_to_free_all_cached_pages(struct erofs_sb_info *sbi,
  287. 	struct erofs_workgroup *egrp);
  288. extern int erofs_try_to_free_cached_page(struct address_space *mapping,
  289. 	struct page *page);
  290. #endif
  291.  
  292. #endif
  293.  
  294. /* we strictly follow PAGE_SIZE and no buffer head yet */
  295. #define LOG_BLOCK_SIZE		PAGE_SHIFT
  296.  
  297. #undef LOG_SECTORS_PER_BLOCK
  298. #define LOG_SECTORS_PER_BLOCK	(PAGE_SHIFT - 9)
  299.  
  300. #undef SECTORS_PER_BLOCK
  301. #define SECTORS_PER_BLOCK	(1 << SECTORS_PER_BLOCK)
  302.  
  303. #define EROFS_BLKSIZ		(1 << LOG_BLOCK_SIZE)
  304.  
  305. #if (EROFS_BLKSIZ % 4096 || !EROFS_BLKSIZ)
  306. #error erofs cannot be used in this platform
  307. #endif
  308.  
  309. #define ROOT_NID(sb)		((sb)->root_nid)
  310.  
  311. #ifdef CONFIG_EROFS_FS_ZIP
  312. /* hard limit of pages per compressed cluster */
  313. #define Z_EROFS_CLUSTER_MAX_PAGES       (CONFIG_EROFS_FS_CLUSTER_PAGE_LIMIT)
  314.  
  315. /* page count of a compressed cluster */
  316. #define erofs_clusterpages(sbi)         ((1 << (sbi)->clusterbits) / PAGE_SIZE)
  317. #endif
  318.  
  319. typedef u64 erofs_off_t;
  320.  
  321. /* data type for filesystem-wide blocks number */
  322. typedef u32 erofs_blk_t;
  323.  
  324. #define erofs_blknr(addr)       ((addr) / EROFS_BLKSIZ)
  325. #define erofs_blkoff(addr)      ((addr) % EROFS_BLKSIZ)
  326. #define blknr_to_addr(nr)       ((erofs_off_t)(nr) * EROFS_BLKSIZ)
  327.  
  328. static inline erofs_off_t iloc(struct erofs_sb_info *sbi, erofs_nid_t nid)
  329. {
  330. 	return blknr_to_addr(sbi->meta_blkaddr) + (nid << sbi->islotbits);
  331. }
  332.  
  333. /* atomic flag definitions */
  334. #define EROFS_V_EA_INITED_BIT	0
  335.  
  336. /* bitlock definitions (arranged in reverse order) */
  337. #define EROFS_V_BL_XATTR_BIT	(BITS_PER_LONG - 1)
  338.  
  339. struct erofs_vnode {
  340. 	erofs_nid_t nid;
  341.  
  342. 	/* atomic flags (including bitlocks) */
  343. 	unsigned long flags;
  344.  
  345. 	unsigned char data_mapping_mode;
  346. 	/* inline size in bytes */
  347. 	unsigned char inode_isize;
  348. 	unsigned short xattr_isize;
  349.  
  350. 	unsigned xattr_shared_count;
  351. 	unsigned *xattr_shared_xattrs;
  352.  
  353. 	erofs_blk_t raw_blkaddr;
  354.  
  355. 	/* the corresponding vfs inode */
  356. 	struct inode vfs_inode;
  357. };
  358.  
  359. #define EROFS_V(ptr)	\
  360. 	container_of(ptr, struct erofs_vnode, vfs_inode)
  361.  
  362. #define __inode_advise(x, bit, bits) \
  363. 	(((x) >> (bit)) & ((1 << (bits)) - 1))
  364.  
  365. #define __inode_version(advise)	\
  366. 	__inode_advise(advise, EROFS_I_VERSION_BIT,	\
  367. 		EROFS_I_VERSION_BITS)
  368.  
  369. #define __inode_data_mapping(advise)	\
  370. 	__inode_advise(advise, EROFS_I_DATA_MAPPING_BIT,\
  371. 		EROFS_I_DATA_MAPPING_BITS)
  372.  
  373. static inline unsigned long inode_datablocks(struct inode *inode)
  374. {
  375. 	/* since i_size cannot be changed */
  376. 	return DIV_ROUND_UP(inode->i_size, EROFS_BLKSIZ);
  377. }
  378.  
  379. static inline bool is_inode_layout_plain(struct inode *inode)
  380. {
  381. 	return EROFS_V(inode)->data_mapping_mode == EROFS_INODE_LAYOUT_PLAIN;
  382. }
  383.  
  384. static inline bool is_inode_layout_compression(struct inode *inode)
  385. {
  386. 	return EROFS_V(inode)->data_mapping_mode ==
  387. 					EROFS_INODE_LAYOUT_COMPRESSION;
  388. }
  389.  
  390. static inline bool is_inode_layout_inline(struct inode *inode)
  391. {
  392. 	return EROFS_V(inode)->data_mapping_mode == EROFS_INODE_LAYOUT_INLINE;
  393. }
  394.  
  395. extern const struct super_operations erofs_sops;
  396. extern const struct inode_operations erofs_dir_iops;
  397. extern const struct file_operations erofs_dir_fops;
  398.  
  399. extern const struct address_space_operations erofs_raw_access_aops;
  400. #ifdef CONFIG_EROFS_FS_ZIP
  401. extern const struct address_space_operations z_erofs_vle_normalaccess_aops;
  402. #endif
  403.  
  404. /*
  405.  * Logical to physical block mapping, used by erofs_map_blocks()
  406.  *
  407.  * Different with other file systems, it is used for 2 access modes:
  408.  *
  409.  * 1) RAW access mode:
  410.  *
  411.  * Users pass a valid (m_lblk, m_lofs -- usually 0) pair,
  412.  * and get the valid m_pblk, m_pofs and the longest m_len(in bytes).
  413.  *
  414.  * Note that m_lblk in the RAW access mode refers to the number of
  415.  * the compressed ondisk block rather than the uncompressed
  416.  * in-memory block for the compressed file.
  417.  *
  418.  * m_pofs equals to m_lofs except for the inline data page.
  419.  *
  420.  * 2) Normal access mode:
  421.  *
  422.  * If the inode is not compressed, it has no difference with
  423.  * the RAW access mode. However, if the inode is compressed,
  424.  * users should pass a valid (m_lblk, m_lofs) pair, and get
  425.  * the needed m_pblk, m_pofs, m_len to get the compressed data
  426.  * and the updated m_lblk, m_lofs which indicates the start
  427.  * of the corresponding uncompressed data in the file.
  428.  */
  429. enum {
  430. 	BH_Zipped = BH_PrivateStart,
  431. };
  432.  
  433. /* Has a disk mapping */
  434. #define EROFS_MAP_MAPPED	(1 << BH_Mapped)
  435. /* Located in metadata (could be copied from bd_inode) */
  436. #define EROFS_MAP_META		(1 << BH_Meta)
  437. /* The extent has been compressed */
  438. #define EROFS_MAP_ZIPPED	(1 << BH_Zipped)
  439.  
  440. struct erofs_map_blocks {
  441. 	erofs_off_t m_pa, m_la;
  442. 	u64 m_plen, m_llen;
  443.  
  444. 	unsigned int m_flags;
  445. };
  446.  
  447. /* Flags used by erofs_map_blocks() */
  448. #define EROFS_GET_BLOCKS_RAW    0x0001
  449.  
  450. /* data.c */
  451. static inline struct bio *prepare_bio(
  452. 	struct super_block *sb,
  453. 	erofs_blk_t blkaddr, unsigned nr_pages,
  454. 	bio_end_io_t endio)
  455. {
  456. 	gfp_t gfp = GFP_NOIO;
  457. 	struct bio *bio = bio_alloc(gfp, nr_pages);
  458.  
  459. 	if (unlikely(bio == NULL) &&
  460. 		(current->flags & PF_MEMALLOC)) {
  461. 		do {
  462. 			nr_pages /= 2;
  463. 			if (unlikely(!nr_pages)) {
  464. 				bio = bio_alloc(gfp | __GFP_NOFAIL, 1);
  465. 				BUG_ON(bio == NULL);
  466. 				break;
  467. 			}
  468. 			bio = bio_alloc(gfp, nr_pages);
  469. 		} while (bio == NULL);
  470. 	}
  471.  
  472. 	bio->bi_end_io = endio;
  473. 	bio_set_dev(bio, sb->s_bdev);
  474. 	bio->bi_iter.bi_sector = blkaddr << LOG_SECTORS_PER_BLOCK;
  475. 	return bio;
  476. }
  477.  
  478. static inline void __submit_bio(struct bio *bio, unsigned op, unsigned op_flags)
  479. {
  480. 	bio_set_op_attrs(bio, op, op_flags);
  481. 	submit_bio(bio);
  482. }
  483.  
  484. extern struct page *erofs_get_meta_page(struct super_block *sb,
  485. 	erofs_blk_t blkaddr, bool prio);
  486. extern int erofs_map_blocks(struct inode *, struct erofs_map_blocks *, int);
  487. extern int erofs_map_blocks_iter(struct inode *, struct erofs_map_blocks *,
  488. 	struct page **, int);
  489.  
  490. struct erofs_map_blocks_iter {
  491. 	struct erofs_map_blocks map;
  492. 	struct page *mpage;
  493. };
  494.  
  495.  
  496. static inline struct page *
  497. erofs_get_inline_page(struct inode *inode,
  498. 		      erofs_blk_t blkaddr)
  499. {
  500. 	return erofs_get_meta_page(inode->i_sb,
  501. 		blkaddr, S_ISDIR(inode->i_mode));
  502. }
  503.  
  504. /* inode.c */
  505. extern struct inode *erofs_iget(struct super_block *sb,
  506. 	erofs_nid_t nid, bool dir);
  507.  
  508. /* dir.c */
  509. int erofs_namei(struct inode *dir, struct qstr *name,
  510. 	erofs_nid_t *nid, unsigned *d_type);
  511.  
  512. /* xattr.c */
  513. #ifdef CONFIG_EROFS_FS_XATTR
  514. extern const struct xattr_handler *erofs_xattr_handlers[];
  515. #endif
  516.  
  517. /* symlink */
  518. #ifdef CONFIG_EROFS_FS_XATTR
  519. extern const struct inode_operations erofs_symlink_xattr_iops;
  520. extern const struct inode_operations erofs_fast_symlink_xattr_iops;
  521. extern const struct inode_operations erofs_special_inode_operations;
  522. #endif
  523.  
  524. static inline void set_inode_fast_symlink(struct inode *inode)
  525. {
  526. #ifdef CONFIG_EROFS_FS_XATTR
  527. 	inode->i_op = &erofs_fast_symlink_xattr_iops;
  528. #else
  529. 	inode->i_op = &simple_symlink_inode_operations;
  530. #endif
  531. }
  532.  
  533. static inline bool is_inode_fast_symlink(struct inode *inode)
  534. {
  535. #ifdef CONFIG_EROFS_FS_XATTR
  536. 	return inode->i_op == &erofs_fast_symlink_xattr_iops;
  537. #else
  538. 	return inode->i_op == &simple_symlink_inode_operations;
  539. #endif
  540. }
  541.  
  542. static inline void *erofs_vmap(struct page **pages, unsigned int count)
  543. {
  544. #ifdef CONFIG_EROFS_FS_USE_VM_MAP_RAM
  545. 	int i = 0;
  546.  
  547. 	while (1) {
  548. 		void *addr = vm_map_ram(pages, count, -1, PAGE_KERNEL);
  549. 		/* retry two more times (totally 3 times) */
  550. 		if (addr != NULL || ++i >= 3)
  551. 			return addr;
  552. 		vm_unmap_aliases();
  553. 	}
  554. 	return NULL;
  555. #else
  556. 	return vmap(pages, count, VM_MAP, PAGE_KERNEL);
  557. #endif
  558. }
  559.  
  560. static inline void erofs_vunmap(const void *mem, unsigned int count)
  561. {
  562. #ifdef CONFIG_EROFS_FS_USE_VM_MAP_RAM
  563. 	vm_unmap_ram(mem, count);
  564. #else
  565. 	vunmap(mem);
  566. #endif
  567. }
  568.  
  569. /* utils.c */
  570. extern struct page *erofs_allocpage(struct list_head *pool, gfp_t gfp);
  571.  
  572. extern void erofs_register_super(struct super_block *sb);
  573. extern void erofs_unregister_super(struct super_block *sb);
  574.  
  575. extern unsigned long erofs_shrink_count(struct shrinker *shrink,
  576. 	struct shrink_control *sc);
  577. extern unsigned long erofs_shrink_scan(struct shrinker *shrink,
  578. 	struct shrink_control *sc);
  579.  
  580. #ifndef lru_to_page
  581. #define lru_to_page(head) (list_entry((head)->prev, struct page, lru))
  582. #endif
  583.  
  584. #endif
  585.

 lz4defs.h

  1. #ifndef __LZ4DEFS_H__
  2. #define __LZ4DEFS_H__
  3.  
  4. /*
  5.  * lz4defs.h -- common and architecture specific defines for the kernel usage
  7.  * LZ4 - Fast LZ compression algorithm
  8.  * Copyright (C) 2011-2016, Yann Collet.
  9.  * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
  10.  * Redistribution and use in source and binary forms, with or without
  11.  * modification, are permitted provided that the following conditions are
  12.  * met:
  13.  *	* Redistributions of source code must retain the above copyright
  14.  *	  notice, this list of conditions and the following disclaimer.
  15.  *	* Redistributions in binary form must reproduce the above
  16.  * copyright notice, this list of conditions and the following disclaimer
  17.  * in the documentation and/or other materials provided with the
  18.  * distribution.
  19.  * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
  20.  * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
  21.  * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
  22.  * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
  23.  * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
  24.  * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
  25.  * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
  26.  * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
  27.  * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
  28.  * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
  29.  * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  30.  * You can contact the author at :
  31.  *	- LZ4 homepage : http://www.lz4.org
  32.  *	- LZ4 source repository : https://github.com/lz4/lz4
  33.  *
  34.  *	Changed for kernel usage by:
  35.  *	Sven Schmidt <4sschmid@informatik.uni-hamburg.de>
  36.  */
  37.  
  38. #include <asm/unaligned.h>
  39. #include <linux/string.h>	 /* memset, memcpy */
  40.  
  41. #define FORCE_INLINE __always_inline
  42.  
  43. /*-************************************
  44.  *	Basic Types
  45.  **************************************/
  46. #include <linux/types.h>
  47.  
  48. typedef	uint8_t BYTE;
  49. typedef uint16_t U16;
  50. typedef uint32_t U32;
  51. typedef	int32_t S32;
  52. typedef uint64_t U64;
  53. typedef uintptr_t uptrval;
  54.  
  55. /*-************************************
  56.  *	Architecture specifics
  57.  **************************************/
  58. #if defined(CONFIG_64BIT)
  59. #define LZ4_ARCH64 1
  60. #else
  61. #define LZ4_ARCH64 0
  62. #endif
  63.  
  64. #if defined(__LITTLE_ENDIAN)
  65. #define LZ4_LITTLE_ENDIAN 1
  66. #else
  67. #define LZ4_LITTLE_ENDIAN 0
  68. #endif
  69.  
  70. /*-************************************
  71.  *	Constants
  72.  **************************************/
  73. #define MINMATCH 4
  74.  
  75. #define WILDCOPYLENGTH 8
  76. #define LASTLITERALS 5
  77. #define MFLIMIT (WILDCOPYLENGTH + MINMATCH)
  78.  
  79. /* Increase this value ==> compression run slower on incompressible data */
  80. #define LZ4_SKIPTRIGGER 6
  81.  
  82. #define HASH_UNIT sizeof(size_t)
  83.  
  84. #define KB (1 << 10)
  85. #define MB (1 << 20)
  86. #define GB (1U << 30)
  87.  
  88. #define MAXD_LOG 16
  89. #define MAX_DISTANCE ((1 << MAXD_LOG) - 1)
  90. #define STEPSIZE sizeof(size_t)
  91.  
  92. #define ML_BITS	4
  93. #define ML_MASK	((1U << ML_BITS) - 1)
  94. #define RUN_BITS (8 - ML_BITS)
  95. #define RUN_MASK ((1U << RUN_BITS) - 1)
  96.  
  97. /*-************************************
  98.  *	Reading and writing into memory
  99.  **************************************/
  100. static FORCE_INLINE U16 LZ4_read16(const void *ptr)
  101. {
  102. 	return get_unaligned((const U16 *)ptr);
  103. }
  104.  
  105. static FORCE_INLINE U32 LZ4_read32(const void *ptr)
  106. {
  107. 	return get_unaligned((const U32 *)ptr);
  108. }
  109.  
  110. static FORCE_INLINE size_t LZ4_read_ARCH(const void *ptr)
  111. {
  112. 	return get_unaligned((const size_t *)ptr);
  113. }
  114.  
  115. static FORCE_INLINE void LZ4_write16(void *memPtr, U16 value)
  116. {
  117. 	put_unaligned(value, (U16 *)memPtr);
  118. }
  119.  
  120. static FORCE_INLINE void LZ4_write32(void *memPtr, U32 value)
  121. {
  122. 	put_unaligned(value, (U32 *)memPtr);
  123. }
  124.  
  125. static FORCE_INLINE U16 LZ4_readLE16(const void *memPtr)
  126. {
  127. 	return get_unaligned_le16(memPtr);
  128. }
  129.  
  130. static FORCE_INLINE void LZ4_writeLE16(void *memPtr, U16 value)
  131. {
  132. 	return put_unaligned_le16(value, memPtr);
  133. }
  134.  
  135. static FORCE_INLINE void LZ4_copy8(void *dst, const void *src)
  136. {
  137. #if LZ4_ARCH64
  138. 	U64 a = get_unaligned((const U64 *)src);
  139.  
  140. 	put_unaligned(a, (U64 *)dst);
  141. #else
  142. 	U32 a = get_unaligned((const U32 *)src);
  143. 	U32 b = get_unaligned((const U32 *)src + 1);
  144.  
  145. 	put_unaligned(a, (U32 *)dst);
  146. 	put_unaligned(b, (U32 *)dst + 1);
  147. #endif
  148. }
  149.  
  150. /*
  151.  * customized variant of memcpy,
  152.  * which can overwrite up to 7 bytes beyond dstEnd
  153.  */
  154. static FORCE_INLINE void LZ4_wildCopy(void *dstPtr,
  155. 	const void *srcPtr, void *dstEnd)
  156. {
  157. 	BYTE *d = (BYTE *)dstPtr;
  158. 	const BYTE *s = (const BYTE *)srcPtr;
  159. 	BYTE *const e = (BYTE *)dstEnd;
  160.  
  161. 	do {
  162. 		LZ4_copy8(d, s);
  163. 		d += 8;
  164. 		s += 8;
  165. 	} while (d < e);
  166. }
  167.  
  168. static FORCE_INLINE unsigned int LZ4_NbCommonBytes(register size_t val)
  169. {
  170. #if LZ4_LITTLE_ENDIAN
  171. 	return __ffs(val) >> 3;
  172. #else
  173. 	return (BITS_PER_LONG - 1 - __fls(val)) >> 3;
  174. #endif
  175. }
  176.  
  177. static FORCE_INLINE unsigned int LZ4_count(
  178. 	const BYTE *pIn,
  179. 	const BYTE *pMatch,
  180. 	const BYTE *pInLimit)
  181. {
  182. 	const BYTE *const pStart = pIn;
  183.  
  184. 	while (likely(pIn < pInLimit - (STEPSIZE - 1))) {
  185. 		size_t const diff = LZ4_read_ARCH(pMatch) ^ LZ4_read_ARCH(pIn);
  186.  
  187. 		if (!diff) {
  188. 			pIn += STEPSIZE;
  189. 			pMatch += STEPSIZE;
  190. 			continue;
  191. 		}
  192.  
  193. 		pIn += LZ4_NbCommonBytes(diff);
  194.  
  195. 		return (unsigned int)(pIn - pStart);
  196. 	}
  197.  
  198. #if LZ4_ARCH64
  199. 	if ((pIn < (pInLimit - 3))
  200. 		&& (LZ4_read32(pMatch) == LZ4_read32(pIn))) {
  201. 		pIn += 4;
  202. 		pMatch += 4;
  203. 	}
  204. #endif
  205.  
  206. 	if ((pIn < (pInLimit - 1))
  207. 		&& (LZ4_read16(pMatch) == LZ4_read16(pIn))) {
  208. 		pIn += 2;
  209. 		pMatch += 2;
  210. 	}
  211.  
  212. 	if ((pIn < pInLimit) && (*pMatch == *pIn))
  213. 		pIn++;
  214.  
  215. 	return (unsigned int)(pIn - pStart);
  216. }
  217.  
  218. typedef enum { noLimit = 0, limitedOutput = 1 } limitedOutput_directive;
  219. typedef enum { byPtr, byU32, byU16 } tableType_t;
  220.  
  221. typedef enum { noDict = 0, withPrefix64k, usingExtDict } dict_directive;
  222. typedef enum { noDictIssue = 0, dictSmall } dictIssue_directive;
  223.  
  224. typedef enum { endOnOutputSize = 0, endOnInputSize = 1 } endCondition_directive;
  225. typedef enum { full = 0, partial = 1 } earlyEnd_directive;
  226.  
  227. #endif

  1. /* SPDX-License-Identifier: GPL-2.0
  2.  *
  3.  * linux/drivers/staging/erofs/xattr.h
  4.  *
  5.  * Copyright (C) 2017-2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #ifndef __EROFS_XATTR_H
  14. #define __EROFS_XATTR_H
  15.  
  16. #include "internal.h"
  17. #include <linux/posix_acl_xattr.h>
  18. #include <linux/xattr.h>
  19.  
  20. /* Attribute not found */
  21. #define ENOATTR         ENODATA
  22.  
  23. static inline unsigned inlinexattr_header_size(struct inode *inode)
  24. {
  25. 	return sizeof(struct erofs_xattr_ibody_header)
  26. 		+ sizeof(u32) * EROFS_V(inode)->xattr_shared_count;
  27. }
  28.  
  29. static inline erofs_blk_t
  30. xattrblock_addr(struct erofs_sb_info *sbi, unsigned xattr_id)
  31. {
  32. #ifdef CONFIG_EROFS_FS_XATTR
  33. 	return sbi->xattr_blkaddr +
  34. 		xattr_id * sizeof(__u32) / EROFS_BLKSIZ;
  35. #else
  36. 	return 0;
  37. #endif
  38. }
  39.  
  40. static inline unsigned
  41. xattrblock_offset(struct erofs_sb_info *sbi, unsigned xattr_id)
  42. {
  43. 	return (xattr_id * sizeof(__u32)) % EROFS_BLKSIZ;
  44. }
  45.  
  46. extern const struct xattr_handler erofs_xattr_user_handler;
  47. extern const struct xattr_handler erofs_xattr_trusted_handler;
  48. #ifdef CONFIG_EROFS_FS_SECURITY
  49. extern const struct xattr_handler erofs_xattr_security_handler;
  50. #endif
  51.  
  52. static inline const struct xattr_handler *erofs_xattr_handler(unsigned index)
  53. {
  54. static const struct xattr_handler *xattr_handler_map[] = {
  55. 	[EROFS_XATTR_INDEX_USER] = &erofs_xattr_user_handler,
  56. #ifdef CONFIG_EROFS_FS_POSIX_ACL
  57. 	[EROFS_XATTR_INDEX_POSIX_ACL_ACCESS] = &posix_acl_access_xattr_handler,
  58. 	[EROFS_XATTR_INDEX_POSIX_ACL_DEFAULT] =
  59. 		&posix_acl_default_xattr_handler,
  60. #endif
  61. 	[EROFS_XATTR_INDEX_TRUSTED] = &erofs_xattr_trusted_handler,
  62. #ifdef CONFIG_EROFS_FS_SECURITY
  63. 	[EROFS_XATTR_INDEX_SECURITY] = &erofs_xattr_security_handler,
  64. #endif
  65. };
  66. 	return index && index < ARRAY_SIZE(xattr_handler_map) ?
  67. 		xattr_handler_map[index] : NULL;
  68. }
  69.  
  70. #ifdef CONFIG_EROFS_FS_XATTR
  71.  
  72. extern const struct inode_operations erofs_generic_xattr_iops;
  73. extern const struct inode_operations erofs_dir_xattr_iops;
  74.  
  75. int erofs_getxattr(struct inode *, int, const char *, void *, size_t);
  76. ssize_t erofs_listxattr(struct dentry *, char *, size_t);
  77. #else
  78. static int __maybe_unused erofs_getxattr(struct inode *inode, int index,
  79. 	const char *name,
  80. 	void *buffer, size_t buffer_size)
  81. {
  82. 	return -ENOTSUPP;
  83. }
  84.  
  85. static ssize_t __maybe_unused erofs_listxattr(struct dentry *dentry,
  86. 	char *buffer, size_t buffer_size)
  87. {
  88. 	return -ENOTSUPP;
  89. }
  90. #endif
  91.  
  92. #endif
  93.

unzip_vle.h

  1. /* SPDX-License-Identifier: GPL-2.0
  2.  *
  3.  * linux/drivers/staging/erofs/unzip_vle.h
  4.  *
  5.  * Copyright (C) 2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #ifndef __EROFS_FS_UNZIP_VLE_H
  14. #define __EROFS_FS_UNZIP_VLE_H
  15.  
  16. #include "internal.h"
  17. #include "unzip_pagevec.h"
  18.  
  19. /*
  20.  *  - 0x5A110C8D ('sallocated', Z_EROFS_MAPPING_STAGING) -
  21.  * used for temporary allocated pages (via erofs_allocpage),
  22.  * in order to seperate those from NULL mapping (eg. truncated pages)
  23.  */
  24. #define Z_EROFS_MAPPING_STAGING		((void *)0x5A110C8D)
  25.  
  26. #define z_erofs_is_stagingpage(page)	\
  27. 	((page)->mapping == Z_EROFS_MAPPING_STAGING)
  28.  
  29. static inline bool z_erofs_gather_if_stagingpage(struct list_head *page_pool,
  30. 						 struct page *page)
  31. {
  32. 	if (z_erofs_is_stagingpage(page)) {
  33. 		list_add(&page->lru, page_pool);
  34. 		return true;
  35. 	}
  36. 	return false;
  37. }
  38.  
  39. /*
  40.  * Structure fields follow one of the following exclusion rules.
  41.  *
  42.  * I: Modifiable by initialization/destruction paths and read-only
  43.  *    for everyone else.
  44.  *
  45.  */
  46.  
  47. #define Z_EROFS_VLE_INLINE_PAGEVECS     3
  48.  
  49. struct z_erofs_vle_work {
  50. 	struct mutex lock;
  51.  
  52. 	/* I: decompression offset in page */
  53. 	unsigned short pageofs;
  54. 	unsigned short nr_pages;
  55.  
  56. 	/* L: queued pages in pagevec[] */
  57. 	unsigned vcnt;
  58.  
  59. 	union {
  60. 		/* L: pagevec */
  61. 		erofs_vtptr_t pagevec[Z_EROFS_VLE_INLINE_PAGEVECS];
  62. 		struct rcu_head rcu;
  63. 	};
  64. };
  65.  
  66. #define Z_EROFS_VLE_WORKGRP_FMT_PLAIN        0
  67. #define Z_EROFS_VLE_WORKGRP_FMT_LZ4          1
  68. #define Z_EROFS_VLE_WORKGRP_FMT_MASK         1
  69.  
  70. typedef struct z_erofs_vle_workgroup *z_erofs_vle_owned_workgrp_t;
  71.  
  72. struct z_erofs_vle_workgroup {
  73. 	struct erofs_workgroup obj;
  74. 	struct z_erofs_vle_work work;
  75.  
  76. 	/* next owned workgroup */
  77. 	z_erofs_vle_owned_workgrp_t next;
  78.  
  79. 	/* compressed pages (including multi-usage pages) */
  80. 	struct page *compressed_pages[Z_EROFS_CLUSTER_MAX_PAGES];
  81. 	unsigned int llen, flags;
  82. };
  83.  
  84. /* let's avoid the valid 32-bit kernel addresses */
  85.  
  86. /* the chained workgroup has't submitted io (still open) */
  87. #define Z_EROFS_VLE_WORKGRP_TAIL        ((void *)0x5F0ECAFE)
  88. /* the chained workgroup has already submitted io */
  89. #define Z_EROFS_VLE_WORKGRP_TAIL_CLOSED ((void *)0x5F0EDEAD)
  90.  
  91. #define Z_EROFS_VLE_WORKGRP_NIL         (NULL)
  92.  
  93. #define z_erofs_vle_workgrp_fmt(grp)	\
  94. 	((grp)->flags & Z_EROFS_VLE_WORKGRP_FMT_MASK)
  95.  
  96. static inline void z_erofs_vle_set_workgrp_fmt(
  97. 	struct z_erofs_vle_workgroup *grp,
  98. 	unsigned int fmt)
  99. {
  100. 	grp->flags = fmt | (grp->flags & ~Z_EROFS_VLE_WORKGRP_FMT_MASK);
  101. }
  102.  
  103.  
  104. /* definitions if multiref is disabled */
  105. #define z_erofs_vle_grab_primary_work(grp)	(&(grp)->work)
  106. #define z_erofs_vle_grab_work(grp, pageofs)	(&(grp)->work)
  107. #define z_erofs_vle_work_workgroup(wrk, primary)	\
  108. 	((primary) ? container_of(wrk,	\
  109. 		struct z_erofs_vle_workgroup, work) : \
  110. 		({ BUG(); (void *)NULL; }))
  111.  
  112.  
  113. #define Z_EROFS_WORKGROUP_SIZE       sizeof(struct z_erofs_vle_workgroup)
  114.  
  115. struct z_erofs_vle_unzip_io {
  116. 	atomic_t pending_bios;
  117. 	z_erofs_vle_owned_workgrp_t head;
  118.  
  119. 	union {
  120. 		wait_queue_head_t wait;
  121. 		struct work_struct work;
  122. 	} u;
  123. };
  124.  
  125. struct z_erofs_vle_unzip_io_sb {
  126. 	struct z_erofs_vle_unzip_io io;
  127. 	struct super_block *sb;
  128. };
  129.  
  130. #define Z_EROFS_ONLINEPAGE_COUNT_BITS 2
  131. #define Z_EROFS_ONLINEPAGE_COUNT_MASK ((1 << Z_EROFS_ONLINEPAGE_COUNT_BITS) - 1)
  132. #define Z_EROFS_ONLINEPAGE_INDEX_SHIFT  (Z_EROFS_ONLINEPAGE_COUNT_BITS)
  133.  
  134. /*
  135.  * waiters (aka. ongoing_packs): # to unlock the page
  136.  * sub-index: 0 - for partial page, >= 1 full page sub-index
  137.  */
  138. typedef atomic_t z_erofs_onlinepage_t;
  139.  
  140. /* type punning */
  141. union z_erofs_onlinepage_converter {
  142. 	z_erofs_onlinepage_t *o;
  143. 	unsigned long *v;
  144. };
  145.  
  146. static inline unsigned z_erofs_onlinepage_index(struct page *page)
  147. {
  148. 	union z_erofs_onlinepage_converter u;
  149.  
  150. 	BUG_ON(!PagePrivate(page));
  151. 	u.v = &page_private(page);
  152.  
  153. 	return atomic_read(u.o) >> Z_EROFS_ONLINEPAGE_INDEX_SHIFT;
  154. }
  155.  
  156. static inline void z_erofs_onlinepage_init(struct page *page)
  157. {
  158. 	union {
  159. 		z_erofs_onlinepage_t o;
  160. 		unsigned long v;
  161. 	/* keep from being unlocked in advance */
  162. 	} u = { .o = ATOMIC_INIT(1) };
  163.  
  164. 	set_page_private(page, u.v);
  165. 	smp_wmb();
  166. 	SetPagePrivate(page);
  167. }
  168.  
  169. static inline void z_erofs_onlinepage_fixup(struct page *page,
  170. 	uintptr_t index, bool down)
  171. {
  172. 	union z_erofs_onlinepage_converter u = { .v = &page_private(page) };
  173. 	int orig, orig_index, val;
  174.  
  175. repeat:
  176. 	orig = atomic_read(u.o);
  177. 	orig_index = orig >> Z_EROFS_ONLINEPAGE_INDEX_SHIFT;
  178. 	if (orig_index) {
  179. 		if (!index)
  180. 			return;
  181.  
  182. 		DBG_BUGON(orig_index != index);
  183. 	}
  184.  
  185. 	val = (index << Z_EROFS_ONLINEPAGE_INDEX_SHIFT) |
  186. 		((orig & Z_EROFS_ONLINEPAGE_COUNT_MASK) + (unsigned int)down);
  187. 	if (atomic_cmpxchg(u.o, orig, val) != orig)
  188. 		goto repeat;
  189. }
  190.  
  191. static inline void z_erofs_onlinepage_endio(struct page *page)
  192. {
  193. 	union z_erofs_onlinepage_converter u;
  194. 	unsigned v;
  195.  
  196. 	BUG_ON(!PagePrivate(page));
  197. 	u.v = &page_private(page);
  198.  
  199. 	v = atomic_dec_return(u.o);
  200. 	if (!(v & Z_EROFS_ONLINEPAGE_COUNT_MASK)) {
  201. 		ClearPagePrivate(page);
  202. 		if (!PageError(page))
  203. 			SetPageUptodate(page);
  204. 		unlock_page(page);
  205. 	}
  206.  
  207. 	debugln("%s, page %p value %x", __func__, page, atomic_read(u.o));
  208. }
  209.  
  210. #define Z_EROFS_VLE_VMAP_ONSTACK_PAGES	\
  211. 	min_t(unsigned int, THREAD_SIZE / 8 / sizeof(struct page *), 96U)
  212. #define Z_EROFS_VLE_VMAP_GLOBAL_PAGES	2048
  213.  
  214. /* unzip_vle_lz4.c */
  215. extern int z_erofs_vle_plain_copy(struct page **compressed_pages,
  216. 	unsigned clusterpages, struct page **pages,
  217. 	unsigned nr_pages, unsigned short pageofs);
  218.  
  219. extern int z_erofs_vle_unzip_fast_percpu(struct page **compressed_pages,
  220. 	unsigned clusterpages, struct page **pages,
  221. 	unsigned int outlen, unsigned short pageofs);
  222.  
  223. extern int z_erofs_vle_unzip_vmap(struct page **compressed_pages,
  224. 	unsigned clusterpages, void *vaddr, unsigned llen,
  225. 	unsigned short pageofs, bool overlapped);
  226.  
  227. #endif
  228.

xattr.h
 

  1. /* SPDX-License-Identifier: GPL-2.0
  2.  *
  3.  * linux/drivers/staging/erofs/xattr.h
  4.  *
  5.  * Copyright (C) 2017-2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #ifndef __EROFS_XATTR_H
  14. #define __EROFS_XATTR_H
  15.  
  16. #include "internal.h"
  17. #include <linux/posix_acl_xattr.h>
  18. #include <linux/xattr.h>
  19.  
  20. /* Attribute not found */
  21. #define ENOATTR         ENODATA
  22.  
  23. static inline unsigned inlinexattr_header_size(struct inode *inode)
  24. {
  25. 	return sizeof(struct erofs_xattr_ibody_header)
  26. 		+ sizeof(u32) * EROFS_V(inode)->xattr_shared_count;
  27. }
  28.  
  29. static inline erofs_blk_t
  30. xattrblock_addr(struct erofs_sb_info *sbi, unsigned xattr_id)
  31. {
  32. #ifdef CONFIG_EROFS_FS_XATTR
  33. 	return sbi->xattr_blkaddr +
  34. 		xattr_id * sizeof(__u32) / EROFS_BLKSIZ;
  35. #else
  36. 	return 0;
  37. #endif
  38. }
  39.  
  40. static inline unsigned
  41. xattrblock_offset(struct erofs_sb_info *sbi, unsigned xattr_id)
  42. {
  43. 	return (xattr_id * sizeof(__u32)) % EROFS_BLKSIZ;
  44. }
  45.  
  46. extern const struct xattr_handler erofs_xattr_user_handler;
  47. extern const struct xattr_handler erofs_xattr_trusted_handler;
  48. #ifdef CONFIG_EROFS_FS_SECURITY
  49. extern const struct xattr_handler erofs_xattr_security_handler;
  50. #endif
  51.  
  52. static inline const struct xattr_handler *erofs_xattr_handler(unsigned index)
  53. {
  54. static const struct xattr_handler *xattr_handler_map[] = {
  55. 	[EROFS_XATTR_INDEX_USER] = &erofs_xattr_user_handler,
  56. #ifdef CONFIG_EROFS_FS_POSIX_ACL
  57. 	[EROFS_XATTR_INDEX_POSIX_ACL_ACCESS] = &posix_acl_access_xattr_handler,
  58. 	[EROFS_XATTR_INDEX_POSIX_ACL_DEFAULT] =
  59. 		&posix_acl_default_xattr_handler,
  60. #endif
  61. 	[EROFS_XATTR_INDEX_TRUSTED] = &erofs_xattr_trusted_handler,
  62. #ifdef CONFIG_EROFS_FS_SECURITY
  63. 	[EROFS_XATTR_INDEX_SECURITY] = &erofs_xattr_security_handler,
  64. #endif
  65. };
  66. 	return index && index < ARRAY_SIZE(xattr_handler_map) ?
  67. 		xattr_handler_map[index] : NULL;
  68. }
  69.  
  70. #ifdef CONFIG_EROFS_FS_XATTR
  71.  
  72. extern const struct inode_operations erofs_generic_xattr_iops;
  73. extern const struct inode_operations erofs_dir_xattr_iops;
  74.  
  75. int erofs_getxattr(struct inode *, int, const char *, void *, size_t);
  76. ssize_t erofs_listxattr(struct dentry *, char *, size_t);
  77. #else
  78. static int __maybe_unused erofs_getxattr(struct inode *inode, int index,
  79. 	const char *name,
  80. 	void *buffer, size_t buffer_size)
  81. {
  82. 	return -ENOTSUPP;
  83. }
  84.  
  85. static ssize_t __maybe_unused erofs_listxattr(struct dentry *dentry,
  86. 	char *buffer, size_t buffer_size)
  87. {
  88. 	return -ENOTSUPP;
  89. }
  90. #endif
  91.  
  92. #endif
  93.

data.c

  1. // SPDX-License-Identifier: GPL-2.0
  2. /*
  3.  * linux/drivers/staging/erofs/data.c
  4.  *
  5.  * Copyright (C) 2017-2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #include "internal.h"
  14. #include <linux/prefetch.h>
  15.  
  16. #include <trace/events/erofs.h>
  17.  
  18. static inline void read_endio(struct bio *bio)
  19. {
  20. 	int i;
  21. 	struct bio_vec *bvec;
  22. 	const blk_status_t err = bio->bi_status;
  23.  
  24. 	bio_for_each_segment_all(bvec, bio, i) {
  25. 		struct page *page = bvec->bv_page;
  26.  
  27. 		/* page is already locked */
  28. 		DBG_BUGON(PageUptodate(page));
  29.  
  30. 		if (unlikely(err))
  31. 			SetPageError(page);
  32. 		else
  33. 			SetPageUptodate(page);
  34.  
  35. 		unlock_page(page);
  36. 		/* page could be reclaimed now */
  37. 	}
  38. 	bio_put(bio);
  39. }
  40.  
  41. /* prio -- true is used for dir */
  42. struct page *erofs_get_meta_page(struct super_block *sb,
  43. 	erofs_blk_t blkaddr, bool prio)
  44. {
  45. 	struct inode *bd_inode = sb->s_bdev->bd_inode;
  46. 	struct address_space *mapping = bd_inode->i_mapping;
  47. 	struct page *page;
  48.  
  49. repeat:
  50. 	page = find_or_create_page(mapping, blkaddr,
  51. 	/*
  52. 	 * Prefer looping in the allocator rather than here,
  53. 	 * at least that code knows what it's doing.
  54. 	 */
  55. 		mapping_gfp_constraint(mapping, ~__GFP_FS) | __GFP_NOFAIL);
  56.  
  57. 	BUG_ON(!page || !PageLocked(page));
  58.  
  59. 	if (!PageUptodate(page)) {
  60. 		struct bio *bio;
  61. 		int err;
  62.  
  63. 		bio = prepare_bio(sb, blkaddr, 1, read_endio);
  64. 		err = bio_add_page(bio, page, PAGE_SIZE, 0);
  65. 		BUG_ON(err != PAGE_SIZE);
  66.  
  67. 		__submit_bio(bio, REQ_OP_READ,
  68. 			REQ_META | (prio ? REQ_PRIO : 0));
  69.  
  70. 		lock_page(page);
  71.  
  72. 		/* the page has been truncated by others? */
  73. 		if (unlikely(page->mapping != mapping)) {
  74. 			unlock_page(page);
  75. 			put_page(page);
  76. 			goto repeat;
  77. 		}
  78.  
  79. 		/* more likely a read error */
  80. 		if (unlikely(!PageUptodate(page))) {
  81. 			unlock_page(page);
  82. 			put_page(page);
  83.  
  84. 			page = ERR_PTR(-EIO);
  85. 		}
  86. 	}
  87. 	return page;
  88. }
  89.  
  90. static int erofs_map_blocks_flatmode(struct inode *inode,
  91. 	struct erofs_map_blocks *map,
  92. 	int flags)
  93. {
  94. 	int err = 0;
  95. 	erofs_blk_t nblocks, lastblk;
  96. 	u64 offset = map->m_la;
  97. 	struct erofs_vnode *vi = EROFS_V(inode);
  98.  
  99. 	trace_erofs_map_blocks_flatmode_enter(inode, map, flags);
  100.  
  101. 	nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
  102. 	lastblk = nblocks - is_inode_layout_inline(inode);
  103.  
  104. 	if (unlikely(offset >= inode->i_size)) {
  105. 		/* leave out-of-bound access unmapped */
  106. 		map->m_flags = 0;
  107. 		map->m_plen = 0;
  108. 		goto out;
  109. 	}
  110.  
  111. 	/* there is no hole in flatmode */
  112. 	map->m_flags = EROFS_MAP_MAPPED;
  113.  
  114. 	if (offset < blknr_to_addr(lastblk)) {
  115. 		map->m_pa = blknr_to_addr(vi->raw_blkaddr) + map->m_la;
  116. 		map->m_plen = blknr_to_addr(lastblk) - offset;
  117. 	} else if (is_inode_layout_inline(inode)) {
  118. 		/* 2 - inode inline B: inode, [xattrs], inline last blk... */
  119. 		struct erofs_sb_info *sbi = EROFS_SB(inode->i_sb);
  120.  
  121. 		map->m_pa = iloc(sbi, vi->nid) + vi->inode_isize +
  122. 			vi->xattr_isize + erofs_blkoff(map->m_la);
  123. 		map->m_plen = inode->i_size - offset;
  124.  
  125. 		/* inline data should locate in one meta block */
  126. 		if (erofs_blkoff(map->m_pa) + map->m_plen > PAGE_SIZE) {
  127. 			DBG_BUGON(1);
  128. 			err = -EIO;
  129. 			goto err_out;
  130. 		}
  131.  
  132. 		map->m_flags |= EROFS_MAP_META;
  133. 	} else {
  134. 		errln("internal error @ nid: %llu (size %llu), m_la 0x%llx",
  135. 			vi->nid, inode->i_size, map->m_la);
  136. 		DBG_BUGON(1);
  137. 		err = -EIO;
  138. 		goto err_out;
  139. 	}
  140.  
  141. out:
  142. 	map->m_llen = map->m_plen;
  143.  
  144. err_out:
  145. 	trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
  146. 	return err;
  147. }
  148.  
  149. #ifdef CONFIG_EROFS_FS_ZIP
  150. extern int z_erofs_map_blocks_iter(struct inode *,
  151. 	struct erofs_map_blocks *, struct page **, int);
  152. #endif
  153.  
  154. int erofs_map_blocks_iter(struct inode *inode,
  155. 	struct erofs_map_blocks *map,
  156. 	struct page **mpage_ret, int flags)
  157. {
  158. 	/* by default, reading raw data never use erofs_map_blocks_iter */
  159. 	if (unlikely(!is_inode_layout_compression(inode))) {
  160. 		if (*mpage_ret != NULL)
  161. 			put_page(*mpage_ret);
  162. 		*mpage_ret = NULL;
  163.  
  164. 		return erofs_map_blocks(inode, map, flags);
  165. 	}
  166.  
  167. #ifdef CONFIG_EROFS_FS_ZIP
  168. 	return z_erofs_map_blocks_iter(inode, map, mpage_ret, flags);
  169. #else
  170. 	/* data compression is not available */
  171. 	return -ENOTSUPP;
  172. #endif
  173. }
  174.  
  175. int erofs_map_blocks(struct inode *inode,
  176. 	struct erofs_map_blocks *map, int flags)
  177. {
  178. 	if (unlikely(is_inode_layout_compression(inode))) {
  179. 		struct page *mpage = NULL;
  180. 		int err;
  181.  
  182. 		err = erofs_map_blocks_iter(inode, map, &mpage, flags);
  183. 		if (mpage != NULL)
  184. 			put_page(mpage);
  185. 		return err;
  186. 	}
  187. 	return erofs_map_blocks_flatmode(inode, map, flags);
  188. }
  189.  
  190. static inline struct bio *erofs_read_raw_page(
  191. 	struct bio *bio,
  192. 	struct address_space *mapping,
  193. 	struct page *page,
  194. 	erofs_off_t *last_block,
  195. 	unsigned nblocks,
  196. 	bool ra)
  197. {
  198. 	struct inode *inode = mapping->host;
  199. 	erofs_off_t current_block = (erofs_off_t)page->index;
  200. 	int err;
  201.  
  202. 	DBG_BUGON(!nblocks);
  203.  
  204. 	if (PageUptodate(page)) {
  205. 		err = 0;
  206. 		goto has_updated;
  207. 	}
  208.  
  209. 	if (cleancache_get_page(page) == 0) {
  210. 		err = 0;
  211. 		SetPageUptodate(page);
  212. 		goto has_updated;
  213. 	}
  214.  
  215. 	/* note that for readpage case, bio also equals to NULL */
  216. 	if (bio != NULL &&
  217. 		/* not continuous */
  218. 		*last_block + 1 != current_block) {
  219. submit_bio_retry:
  220. 		__submit_bio(bio, REQ_OP_READ, 0);
  221. 		bio = NULL;
  222. 	}
  223.  
  224. 	if (bio == NULL) {
  225. 		struct erofs_map_blocks map = {
  226. 			.m_la = blknr_to_addr(current_block),
  227. 		};
  228. 		erofs_blk_t blknr;
  229. 		unsigned blkoff;
  230.  
  231. 		err = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
  232. 		if (unlikely(err))
  233. 			goto err_out;
  234.  
  235. 		/* zero out the holed page */
  236. 		if (unlikely(!(map.m_flags & EROFS_MAP_MAPPED))) {
  237. 			zero_user_segment(page, 0, PAGE_SIZE);
  238. 			SetPageUptodate(page);
  239.  
  240. 			/* imply err = 0, see erofs_map_blocks */
  241. 			goto has_updated;
  242. 		}
  243.  
  244. 		/* for RAW access mode, m_plen must be equal to m_llen */
  245. 		DBG_BUGON(map.m_plen != map.m_llen);
  246.  
  247. 		blknr = erofs_blknr(map.m_pa);
  248. 		blkoff = erofs_blkoff(map.m_pa);
  249.  
  250. 		/* deal with inline page */
  251. 		if (map.m_flags & EROFS_MAP_META) {
  252. 			void *vsrc, *vto;
  253. 			struct page *ipage;
  254.  
  255. 			DBG_BUGON(map.m_plen > PAGE_SIZE);
  256.  
  257. 			ipage = erofs_get_meta_page(inode->i_sb, blknr, 0);
  258.  
  259. 			if (IS_ERR(ipage)) {
  260. 				err = PTR_ERR(ipage);
  261. 				goto err_out;
  262. 			}
  263.  
  264. 			vsrc = kmap_atomic(ipage);
  265. 			vto = kmap_atomic(page);
  266. 			memcpy(vto, vsrc + blkoff, map.m_plen);
  267. 			memset(vto + map.m_plen, 0, PAGE_SIZE - map.m_plen);
  268. 			kunmap_atomic(vto);
  269. 			kunmap_atomic(vsrc);
  270. 			flush_dcache_page(page);
  271.  
  272. 			SetPageUptodate(page);
  273. 			/* TODO: could we unlock the page earlier? */
  274. 			unlock_page(ipage);
  275. 			put_page(ipage);
  276.  
  277. 			/* imply err = 0, see erofs_map_blocks */
  278. 			goto has_updated;
  279. 		}
  280.  
  281. 		/* pa must be block-aligned for raw reading */
  282. 		DBG_BUGON(erofs_blkoff(map.m_pa));
  283.  
  284. 		/* max # of continuous pages */
  285. 		if (nblocks > DIV_ROUND_UP(map.m_plen, PAGE_SIZE))
  286. 			nblocks = DIV_ROUND_UP(map.m_plen, PAGE_SIZE);
  287. 		if (nblocks > BIO_MAX_PAGES)
  288. 			nblocks = BIO_MAX_PAGES;
  289.  
  290. 		bio = prepare_bio(inode->i_sb, blknr, nblocks, read_endio);
  291. 	}
  292.  
  293. 	err = bio_add_page(bio, page, PAGE_SIZE, 0);
  294. 	/* out of the extent or bio is full */
  295. 	if (err < PAGE_SIZE)
  296. 		goto submit_bio_retry;
  297.  
  298. 	*last_block = current_block;
  299.  
  300. 	/* shift in advance in case of it followed by too many gaps */
  301. 	if (unlikely(bio->bi_vcnt >= bio->bi_max_vecs)) {
  302. 		/* err should reassign to 0 after submitting */
  303. 		err = 0;
  304. 		goto submit_bio_out;
  305. 	}
  306.  
  307. 	return bio;
  308.  
  309. err_out:
  310. 	/* for sync reading, set page error immediately */
  311. 	if (!ra) {
  312. 		SetPageError(page);
  313. 		ClearPageUptodate(page);
  314. 	}
  315. has_updated:
  316. 	unlock_page(page);
  317.  
  318. 	/* if updated manually, continuous pages has a gap */
  319. 	if (bio != NULL)
  320. submit_bio_out:
  321. 		__submit_bio(bio, REQ_OP_READ, 0);
  322.  
  323. 	return unlikely(err) ? ERR_PTR(err) : NULL;
  324. }
  325.  
  326. /*
  327.  * since we dont have write or truncate flows, so no inode
  328.  * locking needs to be held at the moment.
  329.  */
  330. static int erofs_raw_access_readpage(struct file *file, struct page *page)
  331. {
  332. 	erofs_off_t last_block;
  333. 	struct bio *bio;
  334.  
  335. 	trace_erofs_readpage(page, true);
  336.  
  337. 	bio = erofs_read_raw_page(NULL, page->mapping,
  338. 		page, &last_block, 1, false);
  339.  
  340. 	if (IS_ERR(bio))
  341. 		return PTR_ERR(bio);
  342.  
  343. 	DBG_BUGON(bio);	/* since we have only one bio -- must be NULL */
  344. 	return 0;
  345. }
  346.  
  347. static int erofs_raw_access_readpages(struct file *filp,
  348. 	struct address_space *mapping,
  349. 	struct list_head *pages, unsigned nr_pages)
  350. {
  351. 	erofs_off_t last_block;
  352. 	struct bio *bio = NULL;
  353. 	gfp_t gfp = readahead_gfp_mask(mapping);
  354. 	struct page *page = list_last_entry(pages, struct page, lru);
  355.  
  356. 	trace_erofs_readpages(mapping->host, page, nr_pages, true);
  357.  
  358. 	for (; nr_pages; --nr_pages) {
  359. 		page = list_entry(pages->prev, struct page, lru);
  360.  
  361. 		prefetchw(&page->flags);
  362. 		list_del(&page->lru);
  363.  
  364. 		if (!add_to_page_cache_lru(page, mapping, page->index, gfp)) {
  365. 			bio = erofs_read_raw_page(bio, mapping, page,
  366. 				&last_block, nr_pages, true);
  367.  
  368. 			/* all the page errors are ignored when readahead */
  369. 			if (IS_ERR(bio)) {
  370. 				pr_err("%s, readahead error at page %lu of nid %llu\n",
  371. 					__func__, page->index,
  372. 					EROFS_V(mapping->host)->nid);
  373.  
  374. 				bio = NULL;
  375. 			}
  376. 		}
  377.  
  378. 		/* pages could still be locked */
  379. 		put_page(page);
  380. 	}
  381. 	DBG_BUGON(!list_empty(pages));
  382.  
  383. 	/* the rare case (end in gaps) */
  384. 	if (unlikely(bio != NULL))
  385. 		__submit_bio(bio, REQ_OP_READ, 0);
  386. 	return 0;
  387. }
  388.  
  389. /* for uncompressed (aligned) files and raw access for other files */
  390. const struct address_space_operations erofs_raw_access_aops = {
  391. 	.readpage = erofs_raw_access_readpage,
  392. 	.readpages = erofs_raw_access_readpages,
  393. };
  394.

dir.c

  1. // SPDX-License-Identifier: GPL-2.0
  2. /*
  3.  * linux/drivers/staging/erofs/dir.c
  4.  *
  5.  * Copyright (C) 2017-2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #include "internal.h"
  14.  
  15. static const unsigned char erofs_filetype_table[EROFS_FT_MAX] = {
  16. 	[EROFS_FT_UNKNOWN]	= DT_UNKNOWN,
  17. 	[EROFS_FT_REG_FILE]	= DT_REG,
  18. 	[EROFS_FT_DIR]		= DT_DIR,
  19. 	[EROFS_FT_CHRDEV]	= DT_CHR,
  20. 	[EROFS_FT_BLKDEV]	= DT_BLK,
  21. 	[EROFS_FT_FIFO]		= DT_FIFO,
  22. 	[EROFS_FT_SOCK]		= DT_SOCK,
  23. 	[EROFS_FT_SYMLINK]	= DT_LNK,
  24. };
  25.  
  26. static void debug_one_dentry(unsigned char d_type, const char *de_name,
  27. 			     unsigned int de_namelen)
  28. {
  29. #ifdef CONFIG_EROFS_FS_DEBUG
  30. 	/* since the on-disk name could not have the trailing '\0' */
  31. 	unsigned char dbg_namebuf[EROFS_NAME_LEN + 1];
  32.  
  33. 	memcpy(dbg_namebuf, de_name, de_namelen);
  34. 	dbg_namebuf[de_namelen] = '\0';
  35.  
  36. 	debugln("found dirent %s de_len %u d_type %d", dbg_namebuf,
  37. 		de_namelen, d_type);
  38. #endif
  39. }
  40.  
  41. static int erofs_fill_dentries(struct dir_context *ctx,
  42. 	void *dentry_blk, unsigned *ofs,
  43. 	unsigned nameoff, unsigned maxsize)
  44. {
  45. 	struct erofs_dirent *de = dentry_blk;
  46. 	const struct erofs_dirent *end = dentry_blk + nameoff;
  47.  
  48. 	de = dentry_blk + *ofs;
  49. 	while (de < end) {
  50. 		const char *de_name;
  51. 		unsigned int de_namelen;
  52. 		unsigned char d_type;
  53.  
  54. 		if (de->file_type < EROFS_FT_MAX)
  55. 			d_type = erofs_filetype_table[de->file_type];
  56. 		else
  57. 			d_type = DT_UNKNOWN;
  58.  
  59. 		nameoff = le16_to_cpu(de->nameoff);
  60. 		de_name = (char *)dentry_blk + nameoff;
  61.  
  62. 		/* the last dirent in the block? */
  63. 		if (de + 1 >= end)
  64. 			de_namelen = strnlen(de_name, maxsize - nameoff);
  65. 		else
  66. 			de_namelen = le16_to_cpu(de[1].nameoff) - nameoff;
  67.  
  68. 		/* a corrupted entry is found */
  69. 		if (unlikely(nameoff + de_namelen > maxsize ||
  70. 			     de_namelen > EROFS_NAME_LEN)) {
  71. 			DBG_BUGON(1);
  72. 			return -EIO;
  73. 		}
  74.  
  75. 		debug_one_dentry(d_type, de_name, de_namelen);
  76. 		if (!dir_emit(ctx, de_name, de_namelen,
  77. 					le64_to_cpu(de->nid), d_type))
  78. 			/* stoped by some reason */
  79. 			return 1;
  80. 		++de;
  81. 		*ofs += sizeof(struct erofs_dirent);
  82. 	}
  83. 	*ofs = maxsize;
  84. 	return 0;
  85. }
  86.  
  87. static int erofs_readdir(struct file *f, struct dir_context *ctx)
  88. {
  89. 	struct inode *dir = file_inode(f);
  90. 	struct address_space *mapping = dir->i_mapping;
  91. 	const size_t dirsize = i_size_read(dir);
  92. 	unsigned i = ctx->pos / EROFS_BLKSIZ;
  93. 	unsigned ofs = ctx->pos % EROFS_BLKSIZ;
  94. 	int err = 0;
  95. 	bool initial = true;
  96.  
  97. 	while (ctx->pos < dirsize) {
  98. 		struct page *dentry_page;
  99. 		struct erofs_dirent *de;
  100. 		unsigned nameoff, maxsize;
  101.  
  102. 		dentry_page = read_mapping_page(mapping, i, NULL);
  103. 		if (dentry_page == ERR_PTR(-ENOMEM)) {
  104. 			err = -ENOMEM;
  105. 			break;
  106. 		} else if (IS_ERR(dentry_page)) {
  107. 			errln("fail to readdir of logical block %u of nid %llu",
  108. 			      i, EROFS_V(dir)->nid);
  109. 			err = PTR_ERR(dentry_page);
  110. 			break;
  111. 		}
  112.  
  113. 		lock_page(dentry_page);
  114. 		de = (struct erofs_dirent *)kmap(dentry_page);
  115.  
  116. 		nameoff = le16_to_cpu(de->nameoff);
  117.  
  118. 		if (unlikely(nameoff < sizeof(struct erofs_dirent) ||
  119. 			nameoff >= PAGE_SIZE)) {
  120. 			errln("%s, invalid de[0].nameoff %u",
  121. 				__func__, nameoff);
  122.  
  123. 			err = -EIO;
  124. 			goto skip_this;
  125. 		}
  126.  
  127. 		maxsize = min_t(unsigned, dirsize - ctx->pos + ofs, PAGE_SIZE);
  128.  
  129. 		/* search dirents at the arbitrary position */
  130. 		if (unlikely(initial)) {
  131. 			initial = false;
  132.  
  133. 			ofs = roundup(ofs, sizeof(struct erofs_dirent));
  134. 			if (unlikely(ofs >= nameoff))
  135. 				goto skip_this;
  136. 		}
  137.  
  138. 		err = erofs_fill_dentries(ctx, de, &ofs, nameoff, maxsize);
  139. skip_this:
  140. 		kunmap(dentry_page);
  141.  
  142. 		unlock_page(dentry_page);
  143. 		put_page(dentry_page);
  144.  
  145. 		ctx->pos = blknr_to_addr(i) + ofs;
  146.  
  147. 		if (unlikely(err))
  148. 			break;
  149. 		++i;
  150. 		ofs = 0;
  151. 	}
  152. 	return err < 0 ? err : 0;
  153. }
  154.  
  155. const struct file_operations erofs_dir_fops = {
  156. 	.llseek		= generic_file_llseek,
  157. 	.read		= generic_read_dir,
  158. 	.iterate	= erofs_readdir,
  159. };
  160.

inode.c

  1. // SPDX-License-Identifier: GPL-2.0
  2. /*
  3.  * linux/drivers/staging/erofs/inode.c
  4.  *
  5.  * Copyright (C) 2017-2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #include "xattr.h"
  14.  
  15. #include <trace/events/erofs.h>
  16.  
  17. /*
  18.  * if inode is successfully read, return its inode page (or sometimes
  19.  * the inode payload page if it's an extended inode) in order to fill
  20.  * inline data if possible.
  21.  */
  22. static struct page *read_inode(struct inode *inode, unsigned int *ofs)
  23. {
  24. 	struct super_block *sb = inode->i_sb;
  25. 	struct erofs_sb_info *sbi = EROFS_SB(sb);
  26. 	struct erofs_vnode *vi = EROFS_V(inode);
  27. 	const erofs_off_t inode_loc = iloc(sbi, vi->nid);
  28. 	erofs_blk_t blkaddr;
  29. 	struct page *page;
  30. 	struct erofs_inode_v1 *v1;
  31. 	struct erofs_inode_v2 *v2, *copied = NULL;
  32. 	unsigned int ifmt;
  33. 	int err;
  34.  
  35. 	blkaddr = erofs_blknr(inode_loc);
  36. 	*ofs = erofs_blkoff(inode_loc);
  37.  
  38. 	debugln("%s, reading inode nid %llu at %u of blkaddr %u",
  39. 		__func__, vi->nid, *ofs, blkaddr);
  40.  
  41. 	page = erofs_get_meta_page(sb, blkaddr, false);
  42. 	if (IS_ERR(page)) {
  43. 		errln("failed to get inode (nid: %llu) page, err %ld",
  44. 		      vi->nid, PTR_ERR(page));
  45. 		return page;
  46. 	}
  47.  
  48. 	v1 = page_address(page) + *ofs;
  49. 	ifmt = le16_to_cpu(v1->i_advise);
  50.  
  51. 	if (ifmt & ~EROFS_I_ALL) {
  52. 		errln("unsupported i_format %u of nid %llu", ifmt, vi->nid);
  53. 		err = -EOPNOTSUPP;
  54. 		goto err_out;
  55. 	}
  56.  
  57. 	vi->data_mapping_mode = __inode_data_mapping(ifmt);
  58. 	if (unlikely(vi->data_mapping_mode >= EROFS_INODE_LAYOUT_MAX)) {
  59. 		errln("unknown data mapping mode %u of nid %llu",
  60. 			vi->data_mapping_mode, vi->nid);
  61. 		err = -EOPNOTSUPP;
  62. 		goto err_out;
  63. 	}
  64.  
  65. 	switch (__inode_version(ifmt)) {
  66. 	case EROFS_INODE_LAYOUT_V2:
  67. 		vi->inode_isize = sizeof(struct erofs_inode_v2);
  68. 		/* check if the inode acrosses page boundary */
  69. 		if (*ofs + vi->inode_isize <= PAGE_SIZE) {
  70. 			*ofs += vi->inode_isize;
  71. 			v2 = (struct erofs_inode_v2 *)v1;
  72. 		} else {
  73. 			const unsigned int gotten = PAGE_SIZE - *ofs;
  74.  
  75. 			copied = kmalloc(vi->inode_isize, GFP_NOFS);
  76. 			if (!copied) {
  77. 				err = -ENOMEM;
  78. 				goto err_out;
  79. 			}
  80. 			memcpy(copied, v1, gotten);
  81. 			unlock_page(page);
  82. 			put_page(page);
  83.  
  84. 			page = erofs_get_meta_page(sb, blkaddr + 1, false);
  85. 			if (IS_ERR(page)) {
  86. 				errln("failed to get inode payload page (nid: %llu), err %ld",
  87. 				      vi->nid, PTR_ERR(page));
  88. 				kfree(copied);
  89. 				return page;
  90. 			}
  91. 			*ofs = vi->inode_isize - gotten;
  92. 			memcpy((u8 *)copied + gotten, page_address(page), *ofs);
  93. 			v2 = copied;
  94. 		}
  95. 		vi->xattr_isize = ondisk_xattr_ibody_size(v2->i_xattr_icount);
  96.  
  97. 		inode->i_mode = le16_to_cpu(v2->i_mode);
  98. 		if (S_ISREG(inode->i_mode) || S_ISDIR(inode->i_mode) ||
  99. 						S_ISLNK(inode->i_mode)) {
  100. 			vi->raw_blkaddr = le32_to_cpu(v2->i_u.raw_blkaddr);
  101. 		} else if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode)) {
  102. 			inode->i_rdev =
  103. 				new_decode_dev(le32_to_cpu(v2->i_u.rdev));
  104. 		} else if (S_ISFIFO(inode->i_mode) || S_ISSOCK(inode->i_mode)) {
  105. 			inode->i_rdev = 0;
  106. 		} else {
  107. 			goto bogusimode;
  108. 		}
  109.  
  110. 		i_uid_write(inode, le32_to_cpu(v2->i_uid));
  111. 		i_gid_write(inode, le32_to_cpu(v2->i_gid));
  112. 		set_nlink(inode, le32_to_cpu(v2->i_nlink));
  113.  
  114. 		/* extended inode has its own timestamp */
  115. 		inode->i_ctime.tv_sec = le64_to_cpu(v2->i_ctime);
  116. 		inode->i_ctime.tv_nsec = le32_to_cpu(v2->i_ctime_nsec);
  117.  
  118. 		inode->i_size = le64_to_cpu(v2->i_size);
  119. 		kfree(copied);
  120. 		break;
  121. 	case EROFS_INODE_LAYOUT_V1:
  122. 		vi->inode_isize = sizeof(struct erofs_inode_v1);
  123. 		*ofs += vi->inode_isize;
  124. 		vi->xattr_isize = ondisk_xattr_ibody_size(v1->i_xattr_icount);
  125.  
  126. 		inode->i_mode = le16_to_cpu(v1->i_mode);
  127. 		if (S_ISREG(inode->i_mode) || S_ISDIR(inode->i_mode) ||
  128. 						S_ISLNK(inode->i_mode)) {
  129. 			vi->raw_blkaddr = le32_to_cpu(v1->i_u.raw_blkaddr);
  130. 		} else if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode)) {
  131. 			inode->i_rdev =
  132. 				new_decode_dev(le32_to_cpu(v1->i_u.rdev));
  133. 		} else if (S_ISFIFO(inode->i_mode) || S_ISSOCK(inode->i_mode)) {
  134. 			inode->i_rdev = 0;
  135. 		} else {
  136. 			goto bogusimode;
  137. 		}
  138.  
  139. 		i_uid_write(inode, le16_to_cpu(v1->i_uid));
  140. 		i_gid_write(inode, le16_to_cpu(v1->i_gid));
  141. 		set_nlink(inode, le16_to_cpu(v1->i_nlink));
  142.  
  143. 		/* use build time for compact inodes */
  144. 		inode->i_ctime.tv_sec = sbi->build_time;
  145. 		inode->i_ctime.tv_nsec = sbi->build_time_nsec;
  146.  
  147. 		inode->i_size = le32_to_cpu(v1->i_size);
  148. 		break;
  149. 	default:
  150. 		errln("unsupported on-disk inode version %u of nid %llu",
  151. 		      __inode_version(ifmt), vi->nid);
  152. 		err = -EOPNOTSUPP;
  153. 		goto err_out;
  154. 	}
  155.  
  156. 	inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
  157. 	inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
  158. 	inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
  159. 	inode->i_atime.tv_nsec = inode->i_ctime.tv_nsec;
  160.  
  161. 	/* measure inode.i_blocks as the generic filesystem */
  162. 	inode->i_blocks = ((inode->i_size - 1) >> 9) + 1;
  163. 	return page;
  164. bogusimode:
  165. 	errln("bogus i_mode (%o) @ nid %llu", inode->i_mode, vi->nid);
  166. 	err = -EIO;
  167. err_out:
  168. 	DBG_BUGON(1);
  169. 	kfree(copied);
  170. 	unlock_page(page);
  171. 	put_page(page);
  172. 	return ERR_PTR(err);
  173. }
  174.  
  175. /*
  176.  * try_lock can be required since locking order is:
  177.  *   file data(fs_inode)
  178.  *        meta(bd_inode)
  179.  * but the majority of the callers is "iget",
  180.  * in that case we are pretty sure no deadlock since
  181.  * no data operations exist. However I tend to
  182.  * try_lock since it takes no much overhead and
  183.  * will success immediately.
  184.  */
  185. static int fill_inline_data(struct inode *inode, void *data, unsigned m_pofs)
  186. {
  187. 	struct erofs_vnode *vi = EROFS_V(inode);
  188. 	struct erofs_sb_info *sbi = EROFS_I_SB(inode);
  189. 	int mode = vi->data_mapping_mode;
  190.  
  191. 	DBG_BUGON(mode >= EROFS_INODE_LAYOUT_MAX);
  192.  
  193. 	/* should be inode inline C */
  194. 	if (mode != EROFS_INODE_LAYOUT_INLINE)
  195. 		return 0;
  196.  
  197. 	/* fast symlink (following ext4) */
  198. 	if (S_ISLNK(inode->i_mode) && inode->i_size < PAGE_SIZE) {
  199. 		char *lnk = erofs_kmalloc(sbi, inode->i_size + 1, GFP_KERNEL);
  200.  
  201. 		if (unlikely(lnk == NULL))
  202. 			return -ENOMEM;
  203.  
  204. 		m_pofs += vi->xattr_isize;
  205.  
  206. 		/* inline symlink data shouldn't across page boundary as well */
  207. 		if (unlikely(m_pofs + inode->i_size > PAGE_SIZE)) {
  208. 			DBG_BUGON(1);
  209. 			kfree(lnk);
  210. 			return -EIO;
  211. 		}
  212.  
  213. 		/* get in-page inline data */
  214. 		memcpy(lnk, data + m_pofs, inode->i_size);
  215. 		lnk[inode->i_size] = '\0';
  216.  
  217. 		inode->i_link = lnk;
  218. 		set_inode_fast_symlink(inode);
  219. 	}
  220. 	return -EAGAIN;
  221. }
  222.  
  223. static int fill_inode(struct inode *inode, int isdir)
  224. {
  225. 	struct page *page;
  226. 	unsigned int ofs;
  227. 	int err = 0;
  228.  
  229. 	trace_erofs_fill_inode(inode, isdir);
  230.  
  231. 	/* read inode base data from disk */
  232. 	page = read_inode(inode, &ofs);
  233. 	if (IS_ERR(page)) {
  234. 		return PTR_ERR(page);
  235. 	} else {
  236. 		/* setup the new inode */
  237. 		if (S_ISREG(inode->i_mode)) {
  238. #ifdef CONFIG_EROFS_FS_XATTR
  239. 			inode->i_op = &erofs_generic_xattr_iops;
  240. #endif
  241. 			inode->i_fop = &generic_ro_fops;
  242. 		} else if (S_ISDIR(inode->i_mode)) {
  243. 			inode->i_op =
  244. #ifdef CONFIG_EROFS_FS_XATTR
  245. 				&erofs_dir_xattr_iops;
  246. #else
  247. 				&erofs_dir_iops;
  248. #endif
  249. 			inode->i_fop = &erofs_dir_fops;
  250. 		} else if (S_ISLNK(inode->i_mode)) {
  251. 			/* by default, page_get_link is used for symlink */
  252. 			inode->i_op =
  253. #ifdef CONFIG_EROFS_FS_XATTR
  254. 				&erofs_symlink_xattr_iops,
  255. #else
  256. 				&page_symlink_inode_operations;
  257. #endif
  258. 			inode_nohighmem(inode);
  259. 		} else if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode) ||
  260. 			S_ISFIFO(inode->i_mode) || S_ISSOCK(inode->i_mode)) {
  261. #ifdef CONFIG_EROFS_FS_XATTR
  262. 			inode->i_op = &erofs_special_inode_operations;
  263. #endif
  264. 			init_special_inode(inode, inode->i_mode, inode->i_rdev);
  265. 		} else {
  266. 			err = -EIO;
  267. 			goto out_unlock;
  268. 		}
  269.  
  270. 		if (is_inode_layout_compression(inode)) {
  271. #ifdef CONFIG_EROFS_FS_ZIP
  272. 			inode->i_mapping->a_ops =
  273. 				&z_erofs_vle_normalaccess_aops;
  274. #else
  275. 			err = -ENOTSUPP;
  276. #endif
  277. 			goto out_unlock;
  278. 		}
  279.  
  280. 		inode->i_mapping->a_ops = &erofs_raw_access_aops;
  281.  
  282. 		/* fill last page if inline data is available */
  283. 		fill_inline_data(inode, page_address(page), ofs);
  284. 	}
  285.  
  286. out_unlock:
  287. 	unlock_page(page);
  288. 	put_page(page);
  289. 	return err;
  290. }
  291.  
  292. struct inode *erofs_iget(struct super_block *sb,
  293. 	erofs_nid_t nid, bool isdir)
  294. {
  295. 	struct inode *inode = iget_locked(sb, nid);
  296.  
  297. 	if (unlikely(inode == NULL))
  298. 		return ERR_PTR(-ENOMEM);
  299.  
  300. 	if (inode->i_state & I_NEW) {
  301. 		int err;
  302. 		struct erofs_vnode *vi = EROFS_V(inode);
  303. 		vi->nid = nid;
  304.  
  305. 		err = fill_inode(inode, isdir);
  306. 		if (likely(!err))
  307. 			unlock_new_inode(inode);
  308. 		else {
  309. 			iget_failed(inode);
  310. 			inode = ERR_PTR(err);
  311. 		}
  312. 	}
  313. 	return inode;
  314. }
  315.  
  316. #ifdef CONFIG_EROFS_FS_XATTR
  317. const struct inode_operations erofs_generic_xattr_iops = {
  318. 	.listxattr = erofs_listxattr,
  319. };
  320. #endif
  321.  
  322. #ifdef CONFIG_EROFS_FS_XATTR
  323. const struct inode_operations erofs_symlink_xattr_iops = {
  324. 	.get_link = page_get_link,
  325. 	.listxattr = erofs_listxattr,
  326. };
  327. #endif
  328.  
  329. const struct inode_operations erofs_special_inode_operations = {
  330. #ifdef CONFIG_EROFS_FS_XATTR
  331. 	.listxattr = erofs_listxattr,
  332. #endif
  333. };
  334.  
  335. #ifdef CONFIG_EROFS_FS_XATTR
  336. const struct inode_operations erofs_fast_symlink_xattr_iops = {
  337. 	.get_link = simple_get_link,
  338. 	.listxattr = erofs_listxattr,
  339. };
  340. #endif
  341.

namei.c

  1. // SPDX-License-Identifier: GPL-2.0
  2. /*
  3.  * linux/drivers/staging/erofs/namei.c
  4.  *
  5.  * Copyright (C) 2017-2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #include "internal.h"
  14. #include "xattr.h"
  15.  
  16. #include <trace/events/erofs.h>
  17.  
  18. struct erofs_qstr {
  19. 	const unsigned char *name;
  20. 	const unsigned char *end;
  21. };
  22.  
  23. /* based on the end of qn is accurate and it must have the trailing '\0' */
  24. static inline int dirnamecmp(const struct erofs_qstr *qn,
  25. 			     const struct erofs_qstr *qd,
  26. 			     unsigned int *matched)
  27. {
  28. 	unsigned int i = *matched;
  29.  
  30. 	/*
  31. 	 * on-disk error, let's only BUG_ON in the debugging mode.
  32. 	 * otherwise, it will return 1 to just skip the invalid name
  33. 	 * and go on (in consideration of the lookup performance).
  34. 	 */
  35. 	DBG_BUGON(qd->name > qd->end);
  36.  
  37. 	/* qd could not have trailing '\0' */
  38. 	/* However it is absolutely safe if < qd->end */
  39. 	while (qd->name + i < qd->end && qd->name[i] != '\0') {
  40. 		if (qn->name[i] != qd->name[i]) {
  41. 			*matched = i;
  42. 			return qn->name[i] > qd->name[i] ? 1 : -1;
  43. 		}
  44. 		++i;
  45. 	}
  46. 	*matched = i;
  47. 	/* See comments in __d_alloc on the terminating NUL character */
  48. 	return qn->name[i] == '\0' ? 0 : 1;
  49. }
  50.  
  51. #define nameoff_from_disk(off, sz)	(le16_to_cpu(off) & ((sz) - 1))
  52.  
  53. static struct erofs_dirent *find_target_dirent(struct erofs_qstr *name,
  54. 					       u8 *data,
  55. 					       unsigned int dirblksize,
  56. 					       const int ndirents)
  57. {
  58. 	int head, back;
  59. 	unsigned int startprfx, endprfx;
  60. 	struct erofs_dirent *const de = (struct erofs_dirent *)data;
  61.  
  62. 	/* since the 1st dirent has been evaluated previously */
  63. 	head = 1;
  64. 	back = ndirents - 1;
  65. 	startprfx = endprfx = 0;
  66.  
  67. 	while (head <= back) {
  68. 		const int mid = head + (back - head) / 2;
  69. 		const int nameoff = nameoff_from_disk(de[mid].nameoff,
  70. 						      dirblksize);
  71. 		unsigned int matched = min(startprfx, endprfx);
  72. 		struct erofs_qstr dname = {
  73. 			.name = data + nameoff,
  74. 			.end = unlikely(mid >= ndirents - 1) ?
  75. 				data + dirblksize :
  76. 				data + nameoff_from_disk(de[mid + 1].nameoff,
  77. 							 dirblksize)
  78. 		};
  79.  
  80. 		/* string comparison without already matched prefix */
  81. 		int ret = dirnamecmp(name, &dname, &matched);
  82.  
  83. 		if (unlikely(!ret)) {
  84. 			return de + mid;
  85. 		} else if (ret > 0) {
  86. 			head = mid + 1;
  87. 			startprfx = matched;
  88. 		} else {
  89. 			back = mid - 1;
  90. 			endprfx = matched;
  91. 		}
  92. 	}
  93.  
  94. 	return ERR_PTR(-ENOENT);
  95. }
  96.  
  97. static struct page *find_target_block_classic(struct inode *dir,
  98. 					      struct erofs_qstr *name,
  99. 					      int *_ndirents)
  100. {
  101. 	unsigned int startprfx, endprfx;
  102. 	int head, back;
  103. 	struct address_space *const mapping = dir->i_mapping;
  104. 	struct page *candidate = ERR_PTR(-ENOENT);
  105.  
  106. 	startprfx = endprfx = 0;
  107. 	head = 0;
  108. 	back = inode_datablocks(dir) - 1;
  109.  
  110. 	while (head <= back) {
  111. 		const int mid = head + (back - head) / 2;
  112. 		struct page *page = read_mapping_page(mapping, mid, NULL);
  113.  
  114. 		if (!IS_ERR(page)) {
  115. 			struct erofs_dirent *de = kmap_atomic(page);
  116. 			const int nameoff = nameoff_from_disk(de->nameoff,
  117. 							      EROFS_BLKSIZ);
  118. 			const int ndirents = nameoff / sizeof(*de);
  119. 			int diff;
  120. 			unsigned int matched;
  121. 			struct erofs_qstr dname;
  122.  
  123. 			if (unlikely(!ndirents)) {
  124. 				DBG_BUGON(1);
  125. 				kunmap_atomic(de);
  126. 				put_page(page);
  127. 				page = ERR_PTR(-EIO);
  128. 				goto out;
  129. 			}
  130.  
  131. 			matched = min(startprfx, endprfx);
  132.  
  133. 			dname.name = (u8 *)de + nameoff;
  134. 			if (ndirents == 1)
  135. 				dname.end = (u8 *)de + EROFS_BLKSIZ;
  136. 			else
  137. 				dname.end = (u8 *)de +
  138. 					nameoff_from_disk(de[1].nameoff,
  139. 							  EROFS_BLKSIZ);
  140.  
  141. 			/* string comparison without already matched prefix */
  142. 			diff = dirnamecmp(name, &dname, &matched);
  143. 			kunmap_atomic(de);
  144.  
  145. 			if (unlikely(!diff)) {
  146. 				*_ndirents = 0;
  147. 				goto out;
  148. 			} else if (diff > 0) {
  149. 				head = mid + 1;
  150. 				startprfx = matched;
  151.  
  152. 				if (likely(!IS_ERR(candidate)))
  153. 					put_page(candidate);
  154. 				candidate = page;
  155. 				*_ndirents = ndirents;
  156. 			} else {
  157. 				put_page(page);
  158.  
  159. 				back = mid - 1;
  160. 				endprfx = matched;
  161. 			}
  162. 			continue;
  163. 		}
  164. out:		/* free if the candidate is valid */
  165. 		if (!IS_ERR(candidate))
  166. 			put_page(candidate);
  167. 		return page;
  168. 	}
  169. 	return candidate;
  170. }
  171.  
  172. int erofs_namei(struct inode *dir,
  173. 		struct qstr *name,
  174. 		erofs_nid_t *nid, unsigned int *d_type)
  175. {
  176. 	int ndirents;
  177. 	struct page *page;
  178. 	void *data;
  179. 	struct erofs_dirent *de;
  180. 	struct erofs_qstr qn;
  181.  
  182. 	if (unlikely(!dir->i_size))
  183. 		return -ENOENT;
  184.  
  185. 	qn.name = name->name;
  186. 	qn.end = name->name + name->len;
  187.  
  188. 	ndirents = 0;
  189. 	page = find_target_block_classic(dir, &qn, &ndirents);
  190.  
  191. 	if (unlikely(IS_ERR(page)))
  192. 		return PTR_ERR(page);
  193.  
  194. 	data = kmap_atomic(page);
  195. 	/* the target page has been mapped */
  196. 	if (ndirents)
  197. 		de = find_target_dirent(&qn, data, EROFS_BLKSIZ, ndirents);
  198. 	else
  199. 		de = (struct erofs_dirent *)data;
  200.  
  201. 	if (likely(!IS_ERR(de))) {
  202. 		*nid = le64_to_cpu(de->nid);
  203. 		*d_type = de->file_type;
  204. 	}
  205.  
  206. 	kunmap_atomic(data);
  207. 	put_page(page);
  208.  
  209. 	return PTR_ERR_OR_ZERO(de);
  210. }
  211.  
  212. /* NOTE: i_mutex is already held by vfs */
  213. static struct dentry *erofs_lookup(struct inode *dir,
  214. 	struct dentry *dentry, unsigned int flags)
  215. {
  216. 	int err;
  217. 	erofs_nid_t nid;
  218. 	unsigned d_type;
  219. 	struct inode *inode;
  220.  
  221. 	DBG_BUGON(!d_really_is_negative(dentry));
  222. 	/* dentry must be unhashed in lookup, no need to worry about */
  223. 	DBG_BUGON(!d_unhashed(dentry));
  224.  
  225. 	trace_erofs_lookup(dir, dentry, flags);
  226.  
  227. 	/* file name exceeds fs limit */
  228. 	if (unlikely(dentry->d_name.len > EROFS_NAME_LEN))
  229. 		return ERR_PTR(-ENAMETOOLONG);
  230.  
  231. 	/* false uninitialized warnings on gcc 4.8.x */
  232. 	err = erofs_namei(dir, &dentry->d_name, &nid, &d_type);
  233.  
  234. 	if (err == -ENOENT) {
  235. 		/* negative dentry */
  236. 		inode = NULL;
  237. 		goto negative_out;
  238. 	} else if (unlikely(err))
  239. 		return ERR_PTR(err);
  240.  
  241. 	debugln("%s, %s (nid %llu) found, d_type %u", __func__,
  242. 		dentry->d_name.name, nid, d_type);
  243.  
  244. 	inode = erofs_iget(dir->i_sb, nid, d_type == EROFS_FT_DIR);
  245. 	if (IS_ERR(inode))
  246. 		return ERR_CAST(inode);
  247.  
  248. negative_out:
  249. 	return d_splice_alias(inode, dentry);
  250. }
  251.  
  252. const struct inode_operations erofs_dir_iops = {
  253. 	.lookup = erofs_lookup,
  254. };
  255.  
  256. const struct inode_operations erofs_dir_xattr_iops = {
  257. 	.lookup = erofs_lookup,
  258. #ifdef CONFIG_EROFS_FS_XATTR
  259. 	.listxattr = erofs_listxattr,
  260. #endif
  261. };
  262.

super.c

  1. // SPDX-License-Identifier: GPL-2.0
  2. /*
  3.  * linux/drivers/staging/erofs/super.c
  4.  *
  5.  * Copyright (C) 2017-2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #include <linux/module.h>
  14. #include <linux/buffer_head.h>
  15. #include <linux/statfs.h>
  16. #include <linux/parser.h>
  17. #include <linux/seq_file.h>
  18. #include "internal.h"
  19.  
  20. #define CREATE_TRACE_POINTS
  21. #include <trace/events/erofs.h>
  22.  
  23. static struct kmem_cache *erofs_inode_cachep __read_mostly;
  24.  
  25. static void init_once(void *ptr)
  26. {
  27. 	struct erofs_vnode *vi = ptr;
  28.  
  29. 	inode_init_once(&vi->vfs_inode);
  30. }
  31.  
  32. static int erofs_init_inode_cache(void)
  33. {
  34. 	erofs_inode_cachep = kmem_cache_create("erofs_inode",
  35. 		sizeof(struct erofs_vnode), 0,
  36. 		SLAB_RECLAIM_ACCOUNT, init_once);
  37.  
  38. 	return erofs_inode_cachep != NULL ? 0 : -ENOMEM;
  39. }
  40.  
  41. static void erofs_exit_inode_cache(void)
  42. {
  43. 	kmem_cache_destroy(erofs_inode_cachep);
  44. }
  45.  
  46. static struct inode *alloc_inode(struct super_block *sb)
  47. {
  48. 	struct erofs_vnode *vi =
  49. 		kmem_cache_alloc(erofs_inode_cachep, GFP_KERNEL);
  50.  
  51. 	if (vi == NULL)
  52. 		return NULL;
  53.  
  54. 	/* zero out everything except vfs_inode */
  55. 	memset(vi, 0, offsetof(struct erofs_vnode, vfs_inode));
  56. 	return &vi->vfs_inode;
  57. }
  58.  
  59. static void i_callback(struct rcu_head *head)
  60. {
  61. 	struct inode *inode = container_of(head, struct inode, i_rcu);
  62. 	struct erofs_vnode *vi = EROFS_V(inode);
  63.  
  64. 	/* be careful RCU symlink path (see ext4_inode_info->i_data)! */
  65. 	if (is_inode_fast_symlink(inode))
  66. 		kfree(inode->i_link);
  67.  
  68. 	kfree(vi->xattr_shared_xattrs);
  69.  
  70. 	kmem_cache_free(erofs_inode_cachep, vi);
  71. }
  72.  
  73. static void destroy_inode(struct inode *inode)
  74. {
  75. 	call_rcu(&inode->i_rcu, i_callback);
  76. }
  77.  
  78. static bool check_layout_compatibility(struct super_block *sb,
  79. 				       struct erofs_super_block *layout)
  80. {
  81. 	const unsigned int requirements = le32_to_cpu(layout->requirements);
  82.  
  83. 	EROFS_SB(sb)->requirements = requirements;
  84.  
  85. 	/* check if current kernel meets all mandatory requirements */
  86. 	if (requirements & (~EROFS_ALL_REQUIREMENTS)) {
  87. 		errln("unidentified requirements %x, please upgrade kernel version",
  88. 		      requirements & ~EROFS_ALL_REQUIREMENTS);
  89. 		return false;
  90. 	}
  91. 	return true;
  92. }
  93.  
  94. static int superblock_read(struct super_block *sb)
  95. {
  96. 	struct erofs_sb_info *sbi;
  97. 	struct buffer_head *bh;
  98. 	struct erofs_super_block *layout;
  99. 	unsigned blkszbits;
  100. 	int ret;
  101.  
  102. 	bh = sb_bread(sb, 0);
  103.  
  104. 	if (bh == NULL) {
  105. 		errln("cannot read erofs superblock");
  106. 		return -EIO;
  107. 	}
  108.  
  109. 	sbi = EROFS_SB(sb);
  110. 	layout = (struct erofs_super_block *)((u8 *)bh->b_data
  111. 		 + EROFS_SUPER_OFFSET);
  112.  
  113. 	ret = -EINVAL;
  114. 	if (le32_to_cpu(layout->magic) != EROFS_SUPER_MAGIC_V1) {
  115. 		errln("cannot find valid erofs superblock");
  116. 		goto out;
  117. 	}
  118.  
  119. 	blkszbits = layout->blkszbits;
  120. 	/* 9(512 bytes) + LOG_SECTORS_PER_BLOCK == LOG_BLOCK_SIZE */
  121. 	if (unlikely(blkszbits != LOG_BLOCK_SIZE)) {
  122. 		errln("blksize %u isn't supported on this platform",
  123. 			1 << blkszbits);
  124. 		goto out;
  125. 	}
  126.  
  127. 	if (!check_layout_compatibility(sb, layout))
  128. 		goto out;
  129.  
  130. 	sbi->blocks = le32_to_cpu(layout->blocks);
  131. 	sbi->meta_blkaddr = le32_to_cpu(layout->meta_blkaddr);
  132. #ifdef CONFIG_EROFS_FS_XATTR
  133. 	sbi->xattr_blkaddr = le32_to_cpu(layout->xattr_blkaddr);
  134. #endif
  135. 	sbi->islotbits = ffs(sizeof(struct erofs_inode_v1)) - 1;
  136. #ifdef CONFIG_EROFS_FS_ZIP
  137. 	sbi->clusterbits = 12;
  138.  
  139. 	if (1 << (sbi->clusterbits - 12) > Z_EROFS_CLUSTER_MAX_PAGES)
  140. 		errln("clusterbits %u is not supported on this kernel",
  141. 			sbi->clusterbits);
  142. #endif
  143.  
  144. 	sbi->root_nid = le16_to_cpu(layout->root_nid);
  145. 	sbi->inos = le64_to_cpu(layout->inos);
  146.  
  147. 	sbi->build_time = le64_to_cpu(layout->build_time);
  148. 	sbi->build_time_nsec = le32_to_cpu(layout->build_time_nsec);
  149.  
  150. 	memcpy(&sb->s_uuid, layout->uuid, sizeof(layout->uuid));
  151. 	memcpy(sbi->volume_name, layout->volume_name,
  152. 		sizeof(layout->volume_name));
  153.  
  154. 	ret = 0;
  155. out:
  156. 	brelse(bh);
  157. 	return ret;
  158. }
  159.  
  160. #ifdef CONFIG_EROFS_FAULT_INJECTION
  161. char *erofs_fault_name[FAULT_MAX] = {
  162. 	[FAULT_KMALLOC]		= "kmalloc",
  163. };
  164.  
  165. static void erofs_build_fault_attr(struct erofs_sb_info *sbi,
  166. 						unsigned int rate)
  167. {
  168. 	struct erofs_fault_info *ffi = &sbi->fault_info;
  169.  
  170. 	if (rate) {
  171. 		atomic_set(&ffi->inject_ops, 0);
  172. 		ffi->inject_rate = rate;
  173. 		ffi->inject_type = (1 << FAULT_MAX) - 1;
  174. 	} else {
  175. 		memset(ffi, 0, sizeof(struct erofs_fault_info));
  176. 	}
  177. }
  178. #endif
  179.  
  180. static void default_options(struct erofs_sb_info *sbi)
  181. {
  182. #ifdef CONFIG_EROFS_FS_XATTR
  183. 	set_opt(sbi, XATTR_USER);
  184. #endif
  185.  
  186. #ifdef CONFIG_EROFS_FS_POSIX_ACL
  187. 	set_opt(sbi, POSIX_ACL);
  188. #endif
  189. }
  190.  
  191. enum {
  192. 	Opt_user_xattr,
  193. 	Opt_nouser_xattr,
  194. 	Opt_acl,
  195. 	Opt_noacl,
  196. 	Opt_fault_injection,
  197. 	Opt_err
  198. };
  199.  
  200. static match_table_t erofs_tokens = {
  201. 	{Opt_user_xattr, "user_xattr"},
  202. 	{Opt_nouser_xattr, "nouser_xattr"},
  203. 	{Opt_acl, "acl"},
  204. 	{Opt_noacl, "noacl"},
  205. 	{Opt_fault_injection, "fault_injection=%u"},
  206. 	{Opt_err, NULL}
  207. };
  208.  
  209. static int parse_options(struct super_block *sb, char *options)
  210. {
  211. 	substring_t args[MAX_OPT_ARGS];
  212. 	char *p;
  213. 	int arg = 0;
  214.  
  215. 	if (!options)
  216. 		return 0;
  217.  
  218. 	while ((p = strsep(&options, ",")) != NULL) {
  219. 		int token;
  220.  
  221. 		if (!*p)
  222. 			continue;
  223.  
  224. 		args[0].to = args[0].from = NULL;
  225. 		token = match_token(p, erofs_tokens, args);
  226.  
  227. 		switch (token) {
  228. #ifdef CONFIG_EROFS_FS_XATTR
  229. 		case Opt_user_xattr:
  230. 			set_opt(EROFS_SB(sb), XATTR_USER);
  231. 			break;
  232. 		case Opt_nouser_xattr:
  233. 			clear_opt(EROFS_SB(sb), XATTR_USER);
  234. 			break;
  235. #else
  236. 		case Opt_user_xattr:
  237. 			infoln("user_xattr options not supported");
  238. 			break;
  239. 		case Opt_nouser_xattr:
  240. 			infoln("nouser_xattr options not supported");
  241. 			break;
  242. #endif
  243. #ifdef CONFIG_EROFS_FS_POSIX_ACL
  244. 		case Opt_acl:
  245. 			set_opt(EROFS_SB(sb), POSIX_ACL);
  246. 			break;
  247. 		case Opt_noacl:
  248. 			clear_opt(EROFS_SB(sb), POSIX_ACL);
  249. 			break;
  250. #else
  251. 		case Opt_acl:
  252. 			infoln("acl options not supported");
  253. 			break;
  254. 		case Opt_noacl:
  255. 			infoln("noacl options not supported");
  256. 			break;
  257. #endif
  258. 		case Opt_fault_injection:
  259. 			if (args->from && match_int(args, &arg))
  260. 				return -EINVAL;
  261. #ifdef CONFIG_EROFS_FAULT_INJECTION
  262. 			erofs_build_fault_attr(EROFS_SB(sb), arg);
  263. 			set_opt(EROFS_SB(sb), FAULT_INJECTION);
  264. #else
  265. 			infoln("FAULT_INJECTION was not selected");
  266. #endif
  267. 			break;
  268. 		default:
  269. 			errln("Unrecognized mount option \"%s\" "
  270. 					"or missing value", p);
  271. 			return -EINVAL;
  272. 		}
  273. 	}
  274. 	return 0;
  275. }
  276.  
  277. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  278.  
  279. static const struct address_space_operations managed_cache_aops;
  280.  
  281. static int managed_cache_releasepage(struct page *page, gfp_t gfp_mask)
  282. {
  283. 	int ret = 1;	/* 0 - busy */
  284. 	struct address_space *const mapping = page->mapping;
  285.  
  286. 	DBG_BUGON(!PageLocked(page));
  287. 	DBG_BUGON(mapping->a_ops != &managed_cache_aops);
  288.  
  289. 	if (PagePrivate(page))
  290. 		ret = erofs_try_to_free_cached_page(mapping, page);
  291.  
  292. 	return ret;
  293. }
  294.  
  295. static void managed_cache_invalidatepage(struct page *page,
  296. 	unsigned int offset, unsigned int length)
  297. {
  298. 	const unsigned int stop = length + offset;
  299.  
  300. 	DBG_BUGON(!PageLocked(page));
  301.  
  302. 	/* Check for potential overflow in debug mode */
  303. 	DBG_BUGON(stop > PAGE_SIZE || stop < length);
  304.  
  305. 	if (offset == 0 && stop == PAGE_SIZE)
  306. 		while (!managed_cache_releasepage(page, GFP_NOFS))
  307. 			cond_resched();
  308. }
  309.  
  310. static const struct address_space_operations managed_cache_aops = {
  311. 	.releasepage = managed_cache_releasepage,
  312. 	.invalidatepage = managed_cache_invalidatepage,
  313. };
  314.  
  315. static struct inode *erofs_init_managed_cache(struct super_block *sb)
  316. {
  317. 	struct inode *inode = new_inode(sb);
  318.  
  319. 	if (unlikely(inode == NULL))
  320. 		return ERR_PTR(-ENOMEM);
  321.  
  322. 	set_nlink(inode, 1);
  323. 	inode->i_size = OFFSET_MAX;
  324.  
  325. 	inode->i_mapping->a_ops = &managed_cache_aops;
  326. 	mapping_set_gfp_mask(inode->i_mapping,
  327. 			     GFP_NOFS | __GFP_HIGHMEM |
  328. 			     __GFP_MOVABLE |  __GFP_NOFAIL);
  329. 	return inode;
  330. }
  331.  
  332. #endif
  333.  
  334. static int erofs_read_super(struct super_block *sb,
  335. 	const char *dev_name, void *data, int silent)
  336. {
  337. 	struct inode *inode;
  338. 	struct erofs_sb_info *sbi;
  339. 	int err = -EINVAL;
  340.  
  341. 	infoln("read_super, device -> %s", dev_name);
  342. 	infoln("options -> %s", (char *)data);
  343.  
  344. 	if (unlikely(!sb_set_blocksize(sb, EROFS_BLKSIZ))) {
  345. 		errln("failed to set erofs blksize");
  346. 		goto err;
  347. 	}
  348.  
  349. 	sbi = kzalloc(sizeof(struct erofs_sb_info), GFP_KERNEL);
  350. 	if (unlikely(sbi == NULL)) {
  351. 		err = -ENOMEM;
  352. 		goto err;
  353. 	}
  354. 	sb->s_fs_info = sbi;
  355.  
  356. 	err = superblock_read(sb);
  357. 	if (err)
  358. 		goto err_sbread;
  359.  
  360. 	sb->s_magic = EROFS_SUPER_MAGIC;
  361. 	sb->s_flags |= SB_RDONLY | SB_NOATIME;
  362. 	sb->s_maxbytes = MAX_LFS_FILESIZE;
  363. 	sb->s_time_gran = 1;
  364.  
  365. 	sb->s_op = &erofs_sops;
  366.  
  367. #ifdef CONFIG_EROFS_FS_XATTR
  368. 	sb->s_xattr = erofs_xattr_handlers;
  369. #endif
  370.  
  371. 	/* set erofs default mount options */
  372. 	default_options(sbi);
  373.  
  374. 	err = parse_options(sb, data);
  375. 	if (err)
  376. 		goto err_parseopt;
  377.  
  378. 	if (!silent)
  379. 		infoln("root inode @ nid %llu", ROOT_NID(sbi));
  380.  
  381. #ifdef CONFIG_EROFS_FS_ZIP
  382. 	INIT_RADIX_TREE(&sbi->workstn_tree, GFP_ATOMIC);
  383. #endif
  384.  
  385. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  386. 	sbi->managed_cache = erofs_init_managed_cache(sb);
  387. 	if (IS_ERR(sbi->managed_cache)) {
  388. 		err = PTR_ERR(sbi->managed_cache);
  389. 		goto err_init_managed_cache;
  390. 	}
  391. #endif
  392.  
  393. 	/* get the root inode */
  394. 	inode = erofs_iget(sb, ROOT_NID(sbi), true);
  395. 	if (IS_ERR(inode)) {
  396. 		err = PTR_ERR(inode);
  397. 		goto err_iget;
  398. 	}
  399.  
  400. 	if (!S_ISDIR(inode->i_mode)) {
  401. 		errln("rootino(nid %llu) is not a directory(i_mode %o)",
  402. 			ROOT_NID(sbi), inode->i_mode);
  403. 		err = -EINVAL;
  404. 		goto err_isdir;
  405. 	}
  406.  
  407. 	sb->s_root = d_make_root(inode);
  408. 	if (sb->s_root == NULL) {
  409. 		err = -ENOMEM;
  410. 		goto err_makeroot;
  411. 	}
  412.  
  413. 	/* save the device name to sbi */
  414. 	sbi->dev_name = __getname();
  415. 	if (sbi->dev_name == NULL) {
  416. 		err = -ENOMEM;
  417. 		goto err_devname;
  418. 	}
  419.  
  420. 	snprintf(sbi->dev_name, PATH_MAX, "%s", dev_name);
  421. 	sbi->dev_name[PATH_MAX - 1] = '\0';
  422.  
  423. 	erofs_register_super(sb);
  424.  
  425. 	if (!silent)
  426. 		infoln("mounted on %s with opts: %s.", dev_name,
  427. 			(char *)data);
  428. 	return 0;
  429. 	/*
  430. 	 * please add a label for each exit point and use
  431. 	 * the following name convention, thus new features
  432. 	 * can be integrated easily without renaming labels.
  433. 	 */
  434. err_devname:
  435. 	dput(sb->s_root);
  436. err_makeroot:
  437. err_isdir:
  438. 	if (sb->s_root == NULL)
  439. 		iput(inode);
  440. err_iget:
  441. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  442. 	iput(sbi->managed_cache);
  443. err_init_managed_cache:
  444. #endif
  445. err_parseopt:
  446. err_sbread:
  447. 	sb->s_fs_info = NULL;
  448. 	kfree(sbi);
  449. err:
  450. 	return err;
  451. }
  452.  
  453. /*
  454.  * could be triggered after deactivate_locked_super()
  455.  * is called, thus including umount and failed to initialize.
  456.  */
  457. static void erofs_put_super(struct super_block *sb)
  458. {
  459. 	struct erofs_sb_info *sbi = EROFS_SB(sb);
  460.  
  461. 	/* for cases which are failed in "read_super" */
  462. 	if (sbi == NULL)
  463. 		return;
  464.  
  465. 	WARN_ON(sb->s_magic != EROFS_SUPER_MAGIC);
  466.  
  467. 	infoln("unmounted for %s", sbi->dev_name);
  468. 	__putname(sbi->dev_name);
  469.  
  470. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  471. 	iput(sbi->managed_cache);
  472. #endif
  473.  
  474. 	mutex_lock(&sbi->umount_mutex);
  475.  
  476. #ifdef CONFIG_EROFS_FS_ZIP
  477. 	erofs_workstation_cleanup_all(sb);
  478. #endif
  479.  
  480. 	erofs_unregister_super(sb);
  481. 	mutex_unlock(&sbi->umount_mutex);
  482.  
  483. 	kfree(sbi);
  484. 	sb->s_fs_info = NULL;
  485. }
  486.  
  487.  
  488. struct erofs_mount_private {
  489. 	const char *dev_name;
  490. 	char *options;
  491. };
  492.  
  493. /* support mount_bdev() with options */
  494. static int erofs_fill_super(struct super_block *sb,
  495. 	void *_priv, int silent)
  496. {
  497. 	struct erofs_mount_private *priv = _priv;
  498.  
  499. 	return erofs_read_super(sb, priv->dev_name,
  500. 		priv->options, silent);
  501. }
  502.  
  503. static struct dentry *erofs_mount(
  504. 	struct file_system_type *fs_type, int flags,
  505. 	const char *dev_name, void *data)
  506. {
  507. 	struct erofs_mount_private priv = {
  508. 		.dev_name = dev_name,
  509. 		.options = data
  510. 	};
  511.  
  512. 	return mount_bdev(fs_type, flags, dev_name,
  513. 		&priv, erofs_fill_super);
  514. }
  515.  
  516. static void erofs_kill_sb(struct super_block *sb)
  517. {
  518. 	kill_block_super(sb);
  519. }
  520.  
  521. static struct shrinker erofs_shrinker_info = {
  522. 	.scan_objects = erofs_shrink_scan,
  523. 	.count_objects = erofs_shrink_count,
  524. 	.seeks = DEFAULT_SEEKS,
  525. };
  526.  
  527. static struct file_system_type erofs_fs_type = {
  528. 	.owner          = THIS_MODULE,
  529. 	.name           = "erofs",
  530. 	.mount          = erofs_mount,
  531. 	.kill_sb        = erofs_kill_sb,
  532. 	.fs_flags       = FS_REQUIRES_DEV,
  533. };
  534. MODULE_ALIAS_FS("erofs");
  535.  
  536. #ifdef CONFIG_EROFS_FS_ZIP
  537. extern int z_erofs_init_zip_subsystem(void);
  538. extern void z_erofs_exit_zip_subsystem(void);
  539. #endif
  540.  
  541. static int __init erofs_module_init(void)
  542. {
  543. 	int err;
  544.  
  545. 	erofs_check_ondisk_layout_definitions();
  546. 	infoln("initializing erofs " EROFS_VERSION);
  547.  
  548. 	err = erofs_init_inode_cache();
  549. 	if (err)
  550. 		goto icache_err;
  551.  
  552. 	err = register_shrinker(&erofs_shrinker_info);
  553. 	if (err)
  554. 		goto shrinker_err;
  555.  
  556. #ifdef CONFIG_EROFS_FS_ZIP
  557. 	err = z_erofs_init_zip_subsystem();
  558. 	if (err)
  559. 		goto zip_err;
  560. #endif
  561.  
  562. 	err = register_filesystem(&erofs_fs_type);
  563. 	if (err)
  564. 		goto fs_err;
  565.  
  566. 	infoln("successfully to initialize erofs");
  567. 	return 0;
  568.  
  569. fs_err:
  570. #ifdef CONFIG_EROFS_FS_ZIP
  571. 	z_erofs_exit_zip_subsystem();
  572. zip_err:
  573. #endif
  574. 	unregister_shrinker(&erofs_shrinker_info);
  575. shrinker_err:
  576. 	erofs_exit_inode_cache();
  577. icache_err:
  578. 	return err;
  579. }
  580.  
  581. static void __exit erofs_module_exit(void)
  582. {
  583. 	unregister_filesystem(&erofs_fs_type);
  584. #ifdef CONFIG_EROFS_FS_ZIP
  585. 	z_erofs_exit_zip_subsystem();
  586. #endif
  587. 	unregister_shrinker(&erofs_shrinker_info);
  588. 	erofs_exit_inode_cache();
  589. 	infoln("successfully finalize erofs");
  590. }
  591.  
  592. /* get filesystem statistics */
  593. static int erofs_statfs(struct dentry *dentry, struct kstatfs *buf)
  594. {
  595. 	struct super_block *sb = dentry->d_sb;
  596. 	struct erofs_sb_info *sbi = EROFS_SB(sb);
  597. 	u64 id = huge_encode_dev(sb->s_bdev->bd_dev);
  598.  
  599. 	buf->f_type = sb->s_magic;
  600. 	buf->f_bsize = EROFS_BLKSIZ;
  601. 	buf->f_blocks = sbi->blocks;
  602. 	buf->f_bfree = buf->f_bavail = 0;
  603.  
  604. 	buf->f_files = ULLONG_MAX;
  605. 	buf->f_ffree = ULLONG_MAX - sbi->inos;
  606.  
  607. 	buf->f_namelen = EROFS_NAME_LEN;
  608.  
  609. 	buf->f_fsid.val[0] = (u32)id;
  610. 	buf->f_fsid.val[1] = (u32)(id >> 32);
  611. 	return 0;
  612. }
  613.  
  614. static int erofs_show_options(struct seq_file *seq, struct dentry *root)
  615. {
  616. 	struct erofs_sb_info *sbi __maybe_unused = EROFS_SB(root->d_sb);
  617.  
  618. #ifdef CONFIG_EROFS_FS_XATTR
  619. 	if (test_opt(sbi, XATTR_USER))
  620. 		seq_puts(seq, ",user_xattr");
  621. 	else
  622. 		seq_puts(seq, ",nouser_xattr");
  623. #endif
  624. #ifdef CONFIG_EROFS_FS_POSIX_ACL
  625. 	if (test_opt(sbi, POSIX_ACL))
  626. 		seq_puts(seq, ",acl");
  627. 	else
  628. 		seq_puts(seq, ",noacl");
  629. #endif
  630. #ifdef CONFIG_EROFS_FAULT_INJECTION
  631. 	if (test_opt(sbi, FAULT_INJECTION))
  632. 		seq_printf(seq, ",fault_injection=%u",
  633. 				sbi->fault_info.inject_rate);
  634. #endif
  635. 	return 0;
  636. }
  637.  
  638. static int erofs_remount(struct super_block *sb, int *flags, char *data)
  639. {
  640. 	DBG_BUGON(!sb_rdonly(sb));
  641.  
  642. 	*flags |= SB_RDONLY;
  643. 	return 0;
  644. }
  645.  
  646. const struct super_operations erofs_sops = {
  647. 	.put_super = erofs_put_super,
  648. 	.alloc_inode = alloc_inode,
  649. 	.destroy_inode = destroy_inode,
  650. 	.statfs = erofs_statfs,
  651. 	.show_options = erofs_show_options,
  652. 	.remount_fs = erofs_remount,
  653. };
  654.  
  655. module_init(erofs_module_init);
  656. module_exit(erofs_module_exit);
  657.  
  658. MODULE_DESCRIPTION("Enhanced ROM File System");
  659. MODULE_AUTHOR("Gao Xiang, Yu Chao, Miao Xie, CONSUMER BG, HUAWEI Inc.");
  660. MODULE_LICENSE("GPL");
  661.

unzip_lz4.c

  1. // SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
  2. /*
  3.  * linux/drivers/staging/erofs/unzip_lz4.c
  4.  *
  5.  * Copyright (C) 2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * Original code taken from 'linux/lib/lz4/lz4_decompress.c'
  10.  */
  11.  
  12. /*
  13.  * LZ4 - Fast LZ compression algorithm
  14.  * Copyright (C) 2011 - 2016, Yann Collet.
  15.  * BSD 2 - Clause License (http://www.opensource.org/licenses/bsd - license.php)
  16.  * Redistribution and use in source and binary forms, with or without
  17.  * modification, are permitted provided that the following conditions are
  18.  * met:
  19.  *	* Redistributions of source code must retain the above copyright
  20.  *	  notice, this list of conditions and the following disclaimer.
  21.  *	* Redistributions in binary form must reproduce the above
  22.  * copyright notice, this list of conditions and the following disclaimer
  23.  * in the documentation and/or other materials provided with the
  24.  * distribution.
  25.  * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
  26.  * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
  27.  * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
  28.  * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
  29.  * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
  30.  * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
  31.  * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
  32.  * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
  33.  * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
  34.  * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
  35.  * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  36.  * You can contact the author at :
  37.  *	- LZ4 homepage : http://www.lz4.org
  38.  *	- LZ4 source repository : https://github.com/lz4/lz4
  39.  *
  40.  *	Changed for kernel usage by:
  41.  *	Sven Schmidt <4sschmid@informatik.uni-hamburg.de>
  42.  */
  43. #include "internal.h"
  44. #include <asm/unaligned.h>
  45. #include "lz4defs.h"
  46.  
  47. /*
  48.  * no public solution to solve our requirement yet.
  49.  * see: <required buffer size for LZ4_decompress_safe_partial>
  50.  *      https://groups.google.com/forum/#!topic/lz4c/_3kkz5N6n00
  51.  */
  52. static FORCE_INLINE int customized_lz4_decompress_safe_partial(
  53. 	const void * const source,
  54. 	void * const dest,
  55. 	int inputSize,
  56. 	int outputSize)
  57. {
  58. 	/* Local Variables */
  59. 	const BYTE *ip = (const BYTE *) source;
  60. 	const BYTE * const iend = ip + inputSize;
  61.  
  62. 	BYTE *op = (BYTE *) dest;
  63. 	BYTE * const oend = op + outputSize;
  64. 	BYTE *cpy;
  65.  
  66. 	static const unsigned int dec32table[] = { 0, 1, 2, 1, 4, 4, 4, 4 };
  67. 	static const int dec64table[] = { 0, 0, 0, -1, 0, 1, 2, 3 };
  68.  
  69. 	/* Empty output buffer */
  70. 	if (unlikely(outputSize == 0))
  71. 		return ((inputSize == 1) && (*ip == 0)) ? 0 : -1;
  72.  
  73. 	/* Main Loop : decode sequences */
  74. 	while (1) {
  75. 		size_t length;
  76. 		const BYTE *match;
  77. 		size_t offset;
  78.  
  79. 		/* get literal length */
  80. 		unsigned int const token = *ip++;
  81.  
  82. 		length = token>>ML_BITS;
  83.  
  84. 		if (length == RUN_MASK) {
  85. 			unsigned int s;
  86.  
  87. 			do {
  88. 				s = *ip++;
  89. 				length += s;
  90. 			} while ((ip < iend - RUN_MASK) & (s == 255));
  91.  
  92. 			if (unlikely((size_t)(op + length) < (size_t)(op))) {
  93. 				/* overflow detection */
  94. 				goto _output_error;
  95. 			}
  96. 			if (unlikely((size_t)(ip + length) < (size_t)(ip))) {
  97. 				/* overflow detection */
  98. 				goto _output_error;
  99. 			}
  100. 		}
  101.  
  102. 		/* copy literals */
  103. 		cpy = op + length;
  104. 		if ((cpy > oend - WILDCOPYLENGTH) ||
  105. 			(ip + length > iend - (2 + 1 + LASTLITERALS))) {
  106. 			if (cpy > oend) {
  107. 				memcpy(op, ip, length = oend - op);
  108. 				op += length;
  109. 				break;
  110. 			}
  111.  
  112. 			if (unlikely(ip + length > iend)) {
  113. 				/*
  114. 				 * Error :
  115. 				 * read attempt beyond
  116. 				 * end of input buffer
  117. 				 */
  118. 				goto _output_error;
  119. 			}
  120.  
  121. 			memcpy(op, ip, length);
  122. 			ip += length;
  123. 			op += length;
  124.  
  125. 			if (ip > iend - 2)
  126. 				break;
  127. 			/* Necessarily EOF, due to parsing restrictions */
  128. 			/* break; */
  129. 		} else {
  130. 			LZ4_wildCopy(op, ip, cpy);
  131. 			ip += length;
  132. 			op = cpy;
  133. 		}
  134.  
  135. 		/* get offset */
  136. 		offset = LZ4_readLE16(ip);
  137. 		ip += 2;
  138. 		match = op - offset;
  139.  
  140. 		if (unlikely(match < (const BYTE *)dest)) {
  141. 			/* Error : offset outside buffers */
  142. 			goto _output_error;
  143. 		}
  144.  
  145. 		/* get matchlength */
  146. 		length = token & ML_MASK;
  147. 		if (length == ML_MASK) {
  148. 			unsigned int s;
  149.  
  150. 			do {
  151. 				s = *ip++;
  152.  
  153. 				if (ip > iend - LASTLITERALS)
  154. 					goto _output_error;
  155.  
  156. 				length += s;
  157. 			} while (s == 255);
  158.  
  159. 			if (unlikely((size_t)(op + length) < (size_t)op)) {
  160. 				/* overflow detection */
  161. 				goto _output_error;
  162. 			}
  163. 		}
  164.  
  165. 		length += MINMATCH;
  166.  
  167. 		/* copy match within block */
  168. 		cpy = op + length;
  169.  
  170. 		if (unlikely(cpy >= oend - WILDCOPYLENGTH)) {
  171. 			if (cpy >= oend) {
  172. 				while (op < oend)
  173. 					*op++ = *match++;
  174. 				break;
  175. 			}
  176. 			goto __match;
  177. 		}
  178.  
  179. 		/* costs ~1%; silence an msan warning when offset == 0 */
  180. 		LZ4_write32(op, (U32)offset);
  181.  
  182. 		if (unlikely(offset < 8)) {
  183. 			const int dec64 = dec64table[offset];
  184.  
  185. 			op[0] = match[0];
  186. 			op[1] = match[1];
  187. 			op[2] = match[2];
  188. 			op[3] = match[3];
  189. 			match += dec32table[offset];
  190. 			memcpy(op + 4, match, 4);
  191. 			match -= dec64;
  192. 		} else {
  193. 			LZ4_copy8(op, match);
  194. 			match += 8;
  195. 		}
  196.  
  197. 		op += 8;
  198.  
  199. 		if (unlikely(cpy > oend - 12)) {
  200. 			BYTE * const oCopyLimit = oend - (WILDCOPYLENGTH - 1);
  201.  
  202. 			if (op < oCopyLimit) {
  203. 				LZ4_wildCopy(op, match, oCopyLimit);
  204. 				match += oCopyLimit - op;
  205. 				op = oCopyLimit;
  206. 			}
  207. __match:
  208. 			while (op < cpy)
  209. 				*op++ = *match++;
  210. 		} else {
  211. 			LZ4_copy8(op, match);
  212.  
  213. 			if (length > 16)
  214. 				LZ4_wildCopy(op + 8, match + 8, cpy);
  215. 		}
  216.  
  217. 		op = cpy; /* correction */
  218. 	}
  219. 	DBG_BUGON((void *)ip - source > inputSize);
  220. 	DBG_BUGON((void *)op - dest > outputSize);
  221.  
  222. 	/* Nb of output bytes decoded */
  223. 	return (int) ((void *)op - dest);
  224.  
  225. 	/* Overflow error detected */
  226. _output_error:
  227. 	return -ERANGE;
  228. }
  229.  
  230. int z_erofs_unzip_lz4(void *in, void *out, size_t inlen, size_t outlen)
  231. {
  232. 	int ret = customized_lz4_decompress_safe_partial(in,
  233. 		out, inlen, outlen);
  234.  
  235. 	if (ret >= 0)
  236. 		return ret;
  237.  
  238. 	/*
  239. 	 * LZ4_decompress_safe will return an error code
  240. 	 * (< 0) if decompression failed
  241. 	 */
  242. 	errln("%s, failed to decompress, in[%p, %zu] outlen[%p, %zu]",
  243. 	      __func__, in, inlen, out, outlen);
  244. 	WARN_ON(1);
  245. 	print_hex_dump(KERN_DEBUG, "raw data [in]: ", DUMP_PREFIX_OFFSET,
  246. 		16, 1, in, inlen, true);
  247. 	print_hex_dump(KERN_DEBUG, "raw data [out]: ", DUMP_PREFIX_OFFSET,
  248. 		16, 1, out, outlen, true);
  249. 	return -EIO;
  250. }
  251.

unzip_vle.c

  1. // SPDX-License-Identifier: GPL-2.0
  2. /*
  3.  * linux/drivers/staging/erofs/unzip_vle.c
  4.  *
  5.  * Copyright (C) 2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #include "unzip_vle.h"
  14. #include <linux/prefetch.h>
  15.  
  16. static struct workqueue_struct *z_erofs_workqueue __read_mostly;
  17. static struct kmem_cache *z_erofs_workgroup_cachep __read_mostly;
  18.  
  19. void z_erofs_exit_zip_subsystem(void)
  20. {
  21. 	destroy_workqueue(z_erofs_workqueue);
  22. 	kmem_cache_destroy(z_erofs_workgroup_cachep);
  23. }
  24.  
  25. static inline int init_unzip_workqueue(void)
  26. {
  27. 	const unsigned onlinecpus = num_possible_cpus();
  28.  
  29. 	/*
  30. 	 * we don't need too many threads, limiting threads
  31. 	 * could improve scheduling performance.
  32. 	 */
  33. 	z_erofs_workqueue = alloc_workqueue("erofs_unzipd",
  34. 		WQ_UNBOUND | WQ_HIGHPRI | WQ_CPU_INTENSIVE,
  35. 		onlinecpus + onlinecpus / 4);
  36.  
  37. 	return z_erofs_workqueue != NULL ? 0 : -ENOMEM;
  38. }
  39.  
  40. int z_erofs_init_zip_subsystem(void)
  41. {
  42. 	z_erofs_workgroup_cachep =
  43. 		kmem_cache_create("erofs_compress",
  44. 		Z_EROFS_WORKGROUP_SIZE, 0,
  45. 		SLAB_RECLAIM_ACCOUNT, NULL);
  46.  
  47. 	if (z_erofs_workgroup_cachep != NULL) {
  48. 		if (!init_unzip_workqueue())
  49. 			return 0;
  50.  
  51. 		kmem_cache_destroy(z_erofs_workgroup_cachep);
  52. 	}
  53. 	return -ENOMEM;
  54. }
  55.  
  56. enum z_erofs_vle_work_role {
  57. 	Z_EROFS_VLE_WORK_SECONDARY,
  58. 	Z_EROFS_VLE_WORK_PRIMARY,
  59. 	/*
  60. 	 * The current work was the tail of an exist chain, and the previous
  61. 	 * processed chained works are all decided to be hooked up to it.
  62. 	 * A new chain should be created for the remaining unprocessed works,
  63. 	 * therefore different from Z_EROFS_VLE_WORK_PRIMARY_FOLLOWED,
  64. 	 * the next work cannot reuse the whole page in the following scenario:
  65. 	 *  ________________________________________________________________
  66. 	 * |      tail (partial) page     |       head (partial) page       |
  67. 	 * |  (belongs to the next work)  |  (belongs to the current work)  |
  68. 	 * |_______PRIMARY_FOLLOWED_______|________PRIMARY_HOOKED___________|
  69. 	 */
  70. 	Z_EROFS_VLE_WORK_PRIMARY_HOOKED,
  71. 	/*
  72. 	 * The current work has been linked with the processed chained works,
  73. 	 * and could be also linked with the potential remaining works, which
  74. 	 * means if the processing page is the tail partial page of the work,
  75. 	 * the current work can safely use the whole page (since the next work
  76. 	 * is under control) for in-place decompression, as illustrated below:
  77. 	 *  ________________________________________________________________
  78. 	 * |  tail (partial) page  |          head (partial) page           |
  79. 	 * | (of the current work) |         (of the previous work)         |
  80. 	 * |  PRIMARY_FOLLOWED or  |                                        |
  81. 	 * |_____PRIMARY_HOOKED____|____________PRIMARY_FOLLOWED____________|
  82. 	 *
  83. 	 * [  (*) the above page can be used for the current work itself.  ]
  84. 	 */
  85. 	Z_EROFS_VLE_WORK_PRIMARY_FOLLOWED,
  86. 	Z_EROFS_VLE_WORK_MAX
  87. };
  88.  
  89. struct z_erofs_vle_work_builder {
  90. 	enum z_erofs_vle_work_role role;
  91. 	/*
  92. 	 * 'hosted = false' means that the current workgroup doesn't belong to
  93. 	 * the owned chained workgroups. In the other words, it is none of our
  94. 	 * business to submit this workgroup.
  95. 	 */
  96. 	bool hosted;
  97.  
  98. 	struct z_erofs_vle_workgroup *grp;
  99. 	struct z_erofs_vle_work *work;
  100. 	struct z_erofs_pagevec_ctor vector;
  101.  
  102. 	/* pages used for reading the compressed data */
  103. 	struct page **compressed_pages;
  104. 	unsigned compressed_deficit;
  105. };
  106.  
  107. #define VLE_WORK_BUILDER_INIT()	\
  108. 	{ .work = NULL, .role = Z_EROFS_VLE_WORK_PRIMARY_FOLLOWED }
  109.  
  110. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  111.  
  112. static bool grab_managed_cache_pages(struct address_space *mapping,
  113. 				     erofs_blk_t start,
  114. 				     struct page **compressed_pages,
  115. 				     int clusterblks,
  116. 				     bool reserve_allocation)
  117. {
  118. 	bool noio = true;
  119. 	unsigned int i;
  120.  
  121. 	/* TODO: optimize by introducing find_get_pages_range */
  122. 	for (i = 0; i < clusterblks; ++i) {
  123. 		struct page *page, *found;
  124.  
  125. 		if (READ_ONCE(compressed_pages[i]) != NULL)
  126. 			continue;
  127.  
  128. 		page = found = find_get_page(mapping, start + i);
  129. 		if (found == NULL) {
  130. 			noio = false;
  131. 			if (!reserve_allocation)
  132. 				continue;
  133. 			page = EROFS_UNALLOCATED_CACHED_PAGE;
  134. 		}
  135.  
  136. 		if (NULL == cmpxchg(compressed_pages + i, NULL, page))
  137. 			continue;
  138.  
  139. 		if (found != NULL)
  140. 			put_page(found);
  141. 	}
  142. 	return noio;
  143. }
  144.  
  145. /* called by erofs_shrinker to get rid of all compressed_pages */
  146. int erofs_try_to_free_all_cached_pages(struct erofs_sb_info *sbi,
  147. 				       struct erofs_workgroup *egrp)
  148. {
  149. 	struct z_erofs_vle_workgroup *const grp =
  150. 		container_of(egrp, struct z_erofs_vle_workgroup, obj);
  151. 	struct address_space *const mapping = sbi->managed_cache->i_mapping;
  152. 	const int clusterpages = erofs_clusterpages(sbi);
  153. 	int i;
  154.  
  155. 	/*
  156. 	 * refcount of workgroup is now freezed as 1,
  157. 	 * therefore no need to worry about available decompression users.
  158. 	 */
  159. 	for (i = 0; i < clusterpages; ++i) {
  160. 		struct page *page = grp->compressed_pages[i];
  161.  
  162. 		if (page == NULL || page->mapping != mapping)
  163. 			continue;
  164.  
  165. 		/* block other users from reclaiming or migrating the page */
  166. 		if (!trylock_page(page))
  167. 			return -EBUSY;
  168.  
  169. 		/* barrier is implied in the following 'unlock_page' */
  170. 		WRITE_ONCE(grp->compressed_pages[i], NULL);
  171.  
  172. 		set_page_private(page, 0);
  173. 		ClearPagePrivate(page);
  174.  
  175. 		unlock_page(page);
  176. 		put_page(page);
  177. 	}
  178. 	return 0;
  179. }
  180.  
  181. int erofs_try_to_free_cached_page(struct address_space *mapping,
  182. 				  struct page *page)
  183. {
  184. 	struct erofs_sb_info *const sbi = EROFS_SB(mapping->host->i_sb);
  185. 	const unsigned int clusterpages = erofs_clusterpages(sbi);
  186.  
  187. 	struct z_erofs_vle_workgroup *grp;
  188. 	int ret = 0;	/* 0 - busy */
  189.  
  190. 	/* prevent the workgroup from being freed */
  191. 	rcu_read_lock();
  192. 	grp = (void *)page_private(page);
  193.  
  194. 	if (erofs_workgroup_try_to_freeze(&grp->obj, 1)) {
  195. 		unsigned int i;
  196.  
  197. 		for (i = 0; i < clusterpages; ++i) {
  198. 			if (grp->compressed_pages[i] == page) {
  199. 				WRITE_ONCE(grp->compressed_pages[i], NULL);
  200. 				ret = 1;
  201. 				break;
  202. 			}
  203. 		}
  204. 		erofs_workgroup_unfreeze(&grp->obj, 1);
  205. 	}
  206. 	rcu_read_unlock();
  207.  
  208. 	if (ret) {
  209. 		ClearPagePrivate(page);
  210. 		put_page(page);
  211. 	}
  212. 	return ret;
  213. }
  214. #endif
  215.  
  216. /* page_type must be Z_EROFS_PAGE_TYPE_EXCLUSIVE */
  217. static inline bool try_to_reuse_as_compressed_page(
  218. 	struct z_erofs_vle_work_builder *b,
  219. 	struct page *page)
  220. {
  221. 	while (b->compressed_deficit) {
  222. 		--b->compressed_deficit;
  223. 		if (NULL == cmpxchg(b->compressed_pages++, NULL, page))
  224. 			return true;
  225. 	}
  226.  
  227. 	return false;
  228. }
  229.  
  230. /* callers must be with work->lock held */
  231. static int z_erofs_vle_work_add_page(struct z_erofs_vle_work_builder *builder,
  232. 				     struct page *page,
  233. 				     enum z_erofs_page_type type,
  234. 				     bool pvec_safereuse)
  235. {
  236. 	int ret;
  237.  
  238. 	/* give priority for the compressed data storage */
  239. 	if (builder->role >= Z_EROFS_VLE_WORK_PRIMARY &&
  240. 		type == Z_EROFS_PAGE_TYPE_EXCLUSIVE &&
  241. 		try_to_reuse_as_compressed_page(builder, page))
  242. 		return 0;
  243.  
  244. 	ret = z_erofs_pagevec_ctor_enqueue(&builder->vector, page, type,
  245. 					   pvec_safereuse);
  246. 	builder->work->vcnt += (unsigned)ret;
  247. 	return ret ? 0 : -EAGAIN;
  248. }
  249.  
  250. static enum z_erofs_vle_work_role
  251. try_to_claim_workgroup(struct z_erofs_vle_workgroup *grp,
  252. 		       z_erofs_vle_owned_workgrp_t *owned_head,
  253. 		       bool *hosted)
  254. {
  255. 	DBG_BUGON(*hosted == true);
  256.  
  257. 	/* let's claim these following types of workgroup */
  258. retry:
  259. 	if (grp->next == Z_EROFS_VLE_WORKGRP_NIL) {
  260. 		/* type 1, nil workgroup */
  261. 		if (Z_EROFS_VLE_WORKGRP_NIL != cmpxchg(&grp->next,
  262. 			Z_EROFS_VLE_WORKGRP_NIL, *owned_head))
  263. 			goto retry;
  264.  
  265. 		*owned_head = grp;
  266. 		*hosted = true;
  267. 		/* lucky, I am the followee :) */
  268. 		return Z_EROFS_VLE_WORK_PRIMARY_FOLLOWED;
  269.  
  270. 	} else if (grp->next == Z_EROFS_VLE_WORKGRP_TAIL) {
  271. 		/*
  272. 		 * type 2, link to the end of a existing open chain,
  273. 		 * be careful that its submission itself is governed
  274. 		 * by the original owned chain.
  275. 		 */
  276. 		if (Z_EROFS_VLE_WORKGRP_TAIL != cmpxchg(&grp->next,
  277. 			Z_EROFS_VLE_WORKGRP_TAIL, *owned_head))
  278. 			goto retry;
  279. 		*owned_head = Z_EROFS_VLE_WORKGRP_TAIL;
  280. 		return Z_EROFS_VLE_WORK_PRIMARY_HOOKED;
  281. 	}
  282.  
  283. 	return Z_EROFS_VLE_WORK_PRIMARY; /* :( better luck next time */
  284. }
  285.  
  286. static struct z_erofs_vle_work *
  287. z_erofs_vle_work_lookup(struct super_block *sb,
  288. 			pgoff_t idx, unsigned pageofs,
  289. 			struct z_erofs_vle_workgroup **grp_ret,
  290. 			enum z_erofs_vle_work_role *role,
  291. 			z_erofs_vle_owned_workgrp_t *owned_head,
  292. 			bool *hosted)
  293. {
  294. 	bool tag, primary;
  295. 	struct erofs_workgroup *egrp;
  296. 	struct z_erofs_vle_workgroup *grp;
  297. 	struct z_erofs_vle_work *work;
  298.  
  299. 	egrp = erofs_find_workgroup(sb, idx, &tag);
  300. 	if (egrp == NULL) {
  301. 		*grp_ret = NULL;
  302. 		return NULL;
  303. 	}
  304.  
  305. 	*grp_ret = grp = container_of(egrp,
  306. 		struct z_erofs_vle_workgroup, obj);
  307.  
  308. 	work = z_erofs_vle_grab_work(grp, pageofs);
  309. 	/* if multiref is disabled, `primary' is always true */
  310. 	primary = true;
  311.  
  312. 	if (work->pageofs != pageofs) {
  313. 		DBG_BUGON(1);
  314. 		erofs_workgroup_put(egrp);
  315. 		return ERR_PTR(-EIO);
  316. 	}
  317.  
  318. 	/*
  319. 	 * lock must be taken first to avoid grp->next == NIL between
  320. 	 * claiming workgroup and adding pages:
  321. 	 *                        grp->next != NIL
  322. 	 *   grp->next = NIL
  323. 	 *   mutex_unlock_all
  324. 	 *                        mutex_lock(&work->lock)
  325. 	 *                        add all pages to pagevec
  326. 	 *
  327. 	 * [correct locking case 1]:
  328. 	 *   mutex_lock(grp->work[a])
  329. 	 *   ...
  330. 	 *   mutex_lock(grp->work[b])     mutex_lock(grp->work[c])
  331. 	 *   ...                          *role = SECONDARY
  332. 	 *                                add all pages to pagevec
  333. 	 *                                ...
  334. 	 *                                mutex_unlock(grp->work[c])
  335. 	 *   mutex_lock(grp->work[c])
  336. 	 *   ...
  337. 	 *   grp->next = NIL
  338. 	 *   mutex_unlock_all
  339. 	 *
  340. 	 * [correct locking case 2]:
  341. 	 *   mutex_lock(grp->work[b])
  342. 	 *   ...
  343. 	 *   mutex_lock(grp->work[a])
  344. 	 *   ...
  345. 	 *   mutex_lock(grp->work[c])
  346. 	 *   ...
  347. 	 *   grp->next = NIL
  348. 	 *   mutex_unlock_all
  349. 	 *                                mutex_lock(grp->work[a])
  350. 	 *                                *role = PRIMARY_OWNER
  351. 	 *                                add all pages to pagevec
  352. 	 *                                ...
  353. 	 */
  354. 	mutex_lock(&work->lock);
  355.  
  356. 	*hosted = false;
  357. 	if (!primary)
  358. 		*role = Z_EROFS_VLE_WORK_SECONDARY;
  359. 	else	/* claim the workgroup if possible */
  360. 		*role = try_to_claim_workgroup(grp, owned_head, hosted);
  361. 	return work;
  362. }
  363.  
  364. static struct z_erofs_vle_work *
  365. z_erofs_vle_work_register(struct super_block *sb,
  366. 			  struct z_erofs_vle_workgroup **grp_ret,
  367. 			  struct erofs_map_blocks *map,
  368. 			  pgoff_t index, unsigned pageofs,
  369. 			  enum z_erofs_vle_work_role *role,
  370. 			  z_erofs_vle_owned_workgrp_t *owned_head,
  371. 			  bool *hosted)
  372. {
  373. 	bool newgrp = false;
  374. 	struct z_erofs_vle_workgroup *grp = *grp_ret;
  375. 	struct z_erofs_vle_work *work;
  376.  
  377. 	/* if multiref is disabled, grp should never be nullptr */
  378. 	if (unlikely(grp)) {
  379. 		DBG_BUGON(1);
  380. 		return ERR_PTR(-EINVAL);
  381. 	}
  382.  
  383. 	/* no available workgroup, let's allocate one */
  384. 	grp = kmem_cache_zalloc(z_erofs_workgroup_cachep, GFP_NOFS);
  385. 	if (unlikely(grp == NULL))
  386. 		return ERR_PTR(-ENOMEM);
  387.  
  388. 	grp->obj.index = index;
  389. 	grp->llen = map->m_llen;
  390.  
  391. 	z_erofs_vle_set_workgrp_fmt(grp,
  392. 		(map->m_flags & EROFS_MAP_ZIPPED) ?
  393. 			Z_EROFS_VLE_WORKGRP_FMT_LZ4 :
  394. 			Z_EROFS_VLE_WORKGRP_FMT_PLAIN);
  395. 	atomic_set(&grp->obj.refcount, 1);
  396.  
  397. 	/* new workgrps have been claimed as type 1 */
  398. 	WRITE_ONCE(grp->next, *owned_head);
  399. 	/* primary and followed work for all new workgrps */
  400. 	*role = Z_EROFS_VLE_WORK_PRIMARY_FOLLOWED;
  401. 	/* it should be submitted by ourselves */
  402. 	*hosted = true;
  403.  
  404. 	newgrp = true;
  405. 	work = z_erofs_vle_grab_primary_work(grp);
  406. 	work->pageofs = pageofs;
  407.  
  408. 	mutex_init(&work->lock);
  409.  
  410. 	if (newgrp) {
  411. 		int err = erofs_register_workgroup(sb, &grp->obj, 0);
  412.  
  413. 		if (err) {
  414. 			kmem_cache_free(z_erofs_workgroup_cachep, grp);
  415. 			return ERR_PTR(-EAGAIN);
  416. 		}
  417. 	}
  418.  
  419. 	*owned_head = *grp_ret = grp;
  420.  
  421. 	mutex_lock(&work->lock);
  422. 	return work;
  423. }
  424.  
  425. static inline void __update_workgrp_llen(struct z_erofs_vle_workgroup *grp,
  426. 					 unsigned int llen)
  427. {
  428. 	while (1) {
  429. 		unsigned int orig_llen = grp->llen;
  430.  
  431. 		if (orig_llen >= llen || orig_llen ==
  432. 			cmpxchg(&grp->llen, orig_llen, llen))
  433. 			break;
  434. 	}
  435. }
  436.  
  437. #define builder_is_hooked(builder) \
  438. 	((builder)->role >= Z_EROFS_VLE_WORK_PRIMARY_HOOKED)
  439.  
  440. #define builder_is_followed(builder) \
  441. 	((builder)->role >= Z_EROFS_VLE_WORK_PRIMARY_FOLLOWED)
  442.  
  443. static int z_erofs_vle_work_iter_begin(struct z_erofs_vle_work_builder *builder,
  444. 				       struct super_block *sb,
  445. 				       struct erofs_map_blocks *map,
  446. 				       z_erofs_vle_owned_workgrp_t *owned_head)
  447. {
  448. 	const unsigned clusterpages = erofs_clusterpages(EROFS_SB(sb));
  449. 	const erofs_blk_t index = erofs_blknr(map->m_pa);
  450. 	const unsigned pageofs = map->m_la & ~PAGE_MASK;
  451. 	struct z_erofs_vle_workgroup *grp;
  452. 	struct z_erofs_vle_work *work;
  453.  
  454. 	DBG_BUGON(builder->work != NULL);
  455.  
  456. 	/* must be Z_EROFS_WORK_TAIL or the next chained work */
  457. 	DBG_BUGON(*owned_head == Z_EROFS_VLE_WORKGRP_NIL);
  458. 	DBG_BUGON(*owned_head == Z_EROFS_VLE_WORKGRP_TAIL_CLOSED);
  459.  
  460. 	DBG_BUGON(erofs_blkoff(map->m_pa));
  461.  
  462. repeat:
  463. 	work = z_erofs_vle_work_lookup(sb, index,
  464. 		pageofs, &grp, &builder->role, owned_head, &builder->hosted);
  465. 	if (work != NULL) {
  466. 		__update_workgrp_llen(grp, map->m_llen);
  467. 		goto got_it;
  468. 	}
  469.  
  470. 	work = z_erofs_vle_work_register(sb, &grp, map, index, pageofs,
  471. 		&builder->role, owned_head, &builder->hosted);
  472.  
  473. 	if (unlikely(work == ERR_PTR(-EAGAIN)))
  474. 		goto repeat;
  475.  
  476. 	if (unlikely(IS_ERR(work)))
  477. 		return PTR_ERR(work);
  478. got_it:
  479. 	z_erofs_pagevec_ctor_init(&builder->vector,
  480. 		Z_EROFS_VLE_INLINE_PAGEVECS, work->pagevec, work->vcnt);
  481.  
  482. 	if (builder->role >= Z_EROFS_VLE_WORK_PRIMARY) {
  483. 		/* enable possibly in-place decompression */
  484. 		builder->compressed_pages = grp->compressed_pages;
  485. 		builder->compressed_deficit = clusterpages;
  486. 	} else {
  487. 		builder->compressed_pages = NULL;
  488. 		builder->compressed_deficit = 0;
  489. 	}
  490.  
  491. 	builder->grp = grp;
  492. 	builder->work = work;
  493. 	return 0;
  494. }
  495.  
  496. /*
  497.  * keep in mind that no referenced workgroups will be freed
  498.  * only after a RCU grace period, so rcu_read_lock() could
  499.  * prevent a workgroup from being freed.
  500.  */
  501. static void z_erofs_rcu_callback(struct rcu_head *head)
  502. {
  503. 	struct z_erofs_vle_work *work =	container_of(head,
  504. 		struct z_erofs_vle_work, rcu);
  505. 	struct z_erofs_vle_workgroup *grp =
  506. 		z_erofs_vle_work_workgroup(work, true);
  507.  
  508. 	kmem_cache_free(z_erofs_workgroup_cachep, grp);
  509. }
  510.  
  511. void erofs_workgroup_free_rcu(struct erofs_workgroup *grp)
  512. {
  513. 	struct z_erofs_vle_workgroup *const vgrp = container_of(grp,
  514. 		struct z_erofs_vle_workgroup, obj);
  515. 	struct z_erofs_vle_work *const work = &vgrp->work;
  516.  
  517. 	call_rcu(&work->rcu, z_erofs_rcu_callback);
  518. }
  519.  
  520. static void __z_erofs_vle_work_release(struct z_erofs_vle_workgroup *grp,
  521. 	struct z_erofs_vle_work *work __maybe_unused)
  522. {
  523. 	erofs_workgroup_put(&grp->obj);
  524. }
  525.  
  526. void z_erofs_vle_work_release(struct z_erofs_vle_work *work)
  527. {
  528. 	struct z_erofs_vle_workgroup *grp =
  529. 		z_erofs_vle_work_workgroup(work, true);
  530.  
  531. 	__z_erofs_vle_work_release(grp, work);
  532. }
  533.  
  534. static inline bool
  535. z_erofs_vle_work_iter_end(struct z_erofs_vle_work_builder *builder)
  536. {
  537. 	struct z_erofs_vle_work *work = builder->work;
  538.  
  539. 	if (work == NULL)
  540. 		return false;
  541.  
  542. 	z_erofs_pagevec_ctor_exit(&builder->vector, false);
  543. 	mutex_unlock(&work->lock);
  544.  
  545. 	/*
  546. 	 * if all pending pages are added, don't hold work reference
  547. 	 * any longer if the current work isn't hosted by ourselves.
  548. 	 */
  549. 	if (!builder->hosted)
  550. 		__z_erofs_vle_work_release(builder->grp, work);
  551.  
  552. 	builder->work = NULL;
  553. 	builder->grp = NULL;
  554. 	return true;
  555. }
  556.  
  557. static inline struct page *__stagingpage_alloc(struct list_head *pagepool,
  558. 					       gfp_t gfp)
  559. {
  560. 	struct page *page = erofs_allocpage(pagepool, gfp);
  561.  
  562. 	if (unlikely(page == NULL))
  563. 		return NULL;
  564.  
  565. 	page->mapping = Z_EROFS_MAPPING_STAGING;
  566. 	return page;
  567. }
  568.  
  569. struct z_erofs_vle_frontend {
  570. 	struct inode *const inode;
  571.  
  572. 	struct z_erofs_vle_work_builder builder;
  573. 	struct erofs_map_blocks_iter m_iter;
  574.  
  575. 	z_erofs_vle_owned_workgrp_t owned_head;
  576.  
  577. 	bool initial;
  578. #if (EROFS_FS_ZIP_CACHE_LVL >= 2)
  579. 	erofs_off_t cachedzone_la;
  580. #endif
  581. };
  582.  
  583. #define VLE_FRONTEND_INIT(__i) { \
  584. 	.inode = __i, \
  585. 	.m_iter = { \
  586. 		{ .m_llen = 0, .m_plen = 0 }, \
  587. 		.mpage = NULL \
  588. 	}, \
  589. 	.builder = VLE_WORK_BUILDER_INIT(), \
  590. 	.owned_head = Z_EROFS_VLE_WORKGRP_TAIL, \
  591. 	.initial = true, }
  592.  
  593. static int z_erofs_do_read_page(struct z_erofs_vle_frontend *fe,
  594. 				struct page *page,
  595. 				struct list_head *page_pool)
  596. {
  597. 	struct super_block *const sb = fe->inode->i_sb;
  598. 	struct erofs_sb_info *const sbi __maybe_unused = EROFS_SB(sb);
  599. 	struct erofs_map_blocks_iter *const m = &fe->m_iter;
  600. 	struct erofs_map_blocks *const map = &m->map;
  601. 	struct z_erofs_vle_work_builder *const builder = &fe->builder;
  602. 	const loff_t offset = page_offset(page);
  603.  
  604. 	bool tight = builder_is_hooked(builder);
  605. 	struct z_erofs_vle_work *work = builder->work;
  606.  
  607. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  608. 	struct address_space *const mngda = sbi->managed_cache->i_mapping;
  609. 	struct z_erofs_vle_workgroup *grp;
  610. 	bool noio_outoforder;
  611. #endif
  612.  
  613. 	enum z_erofs_page_type page_type;
  614. 	unsigned cur, end, spiltted, index;
  615. 	int err = 0;
  616.  
  617. 	/* register locked file pages as online pages in pack */
  618. 	z_erofs_onlinepage_init(page);
  619.  
  620. 	spiltted = 0;
  621. 	end = PAGE_SIZE;
  622. repeat:
  623. 	cur = end - 1;
  624.  
  625. 	/* lucky, within the range of the current map_blocks */
  626. 	if (offset + cur >= map->m_la &&
  627. 		offset + cur < map->m_la + map->m_llen) {
  628. 		/* didn't get a valid unzip work previously (very rare) */
  629. 		if (!builder->work)
  630. 			goto restart_now;
  631. 		goto hitted;
  632. 	}
  633.  
  634. 	/* go ahead the next map_blocks */
  635. 	debugln("%s: [out-of-range] pos %llu", __func__, offset + cur);
  636.  
  637. 	if (z_erofs_vle_work_iter_end(builder))
  638. 		fe->initial = false;
  639.  
  640. 	map->m_la = offset + cur;
  641. 	map->m_llen = 0;
  642. 	err = erofs_map_blocks_iter(fe->inode, map, &m->mpage, 0);
  643. 	if (unlikely(err))
  644. 		goto err_out;
  645.  
  646. restart_now:
  647. 	if (unlikely(!(map->m_flags & EROFS_MAP_MAPPED)))
  648. 		goto hitted;
  649.  
  650. 	DBG_BUGON(map->m_plen != 1 << sbi->clusterbits);
  651. 	DBG_BUGON(erofs_blkoff(map->m_pa));
  652.  
  653. 	err = z_erofs_vle_work_iter_begin(builder, sb, map, &fe->owned_head);
  654. 	if (unlikely(err))
  655. 		goto err_out;
  656.  
  657. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  658. 	grp = fe->builder.grp;
  659.  
  660. 	/* let's do out-of-order decompression for noio */
  661. 	noio_outoforder = grab_managed_cache_pages(mngda,
  662. 		erofs_blknr(map->m_pa),
  663. 		grp->compressed_pages, erofs_blknr(map->m_plen),
  664. 		/* compressed page caching selection strategy */
  665. 		fe->initial | (EROFS_FS_ZIP_CACHE_LVL >= 2 ?
  666. 			map->m_la < fe->cachedzone_la : 0));
  667.  
  668. 	if (noio_outoforder && builder_is_followed(builder))
  669. 		builder->role = Z_EROFS_VLE_WORK_PRIMARY;
  670. #endif
  671.  
  672. 	tight &= builder_is_hooked(builder);
  673. 	work = builder->work;
  674. hitted:
  675. 	cur = end - min_t(unsigned, offset + end - map->m_la, end);
  676. 	if (unlikely(!(map->m_flags & EROFS_MAP_MAPPED))) {
  677. 		zero_user_segment(page, cur, end);
  678. 		goto next_part;
  679. 	}
  680.  
  681. 	/* let's derive page type */
  682. 	page_type = cur ? Z_EROFS_VLE_PAGE_TYPE_HEAD :
  683. 		(!spiltted ? Z_EROFS_PAGE_TYPE_EXCLUSIVE :
  684. 			(tight ? Z_EROFS_PAGE_TYPE_EXCLUSIVE :
  685. 				Z_EROFS_VLE_PAGE_TYPE_TAIL_SHARED));
  686.  
  687. 	if (cur)
  688. 		tight &= builder_is_followed(builder);
  689.  
  690. retry:
  691. 	err = z_erofs_vle_work_add_page(builder, page, page_type,
  692. 					builder_is_followed(builder));
  693. 	/* should allocate an additional staging page for pagevec */
  694. 	if (err == -EAGAIN) {
  695. 		struct page *const newpage =
  696. 			__stagingpage_alloc(page_pool, GFP_NOFS);
  697.  
  698. 		err = z_erofs_vle_work_add_page(builder,
  699. 			newpage, Z_EROFS_PAGE_TYPE_EXCLUSIVE, true);
  700. 		if (likely(!err))
  701. 			goto retry;
  702. 	}
  703.  
  704. 	if (unlikely(err))
  705. 		goto err_out;
  706.  
  707. 	index = page->index - map->m_la / PAGE_SIZE;
  708.  
  709. 	/* FIXME! avoid the last relundant fixup & endio */
  710. 	z_erofs_onlinepage_fixup(page, index, true);
  711.  
  712. 	/* bump up the number of spiltted parts of a page */
  713. 	++spiltted;
  714. 	/* also update nr_pages */
  715. 	work->nr_pages = max_t(pgoff_t, work->nr_pages, index + 1);
  716. next_part:
  717. 	/* can be used for verification */
  718. 	map->m_llen = offset + cur - map->m_la;
  719.  
  720. 	end = cur;
  721. 	if (end > 0)
  722. 		goto repeat;
  723.  
  724. out:
  725. 	/* FIXME! avoid the last relundant fixup & endio */
  726. 	z_erofs_onlinepage_endio(page);
  727.  
  728. 	debugln("%s, finish page: %pK spiltted: %u map->m_llen %llu",
  729. 		__func__, page, spiltted, map->m_llen);
  730. 	return err;
  731.  
  732. 	/* if some error occurred while processing this page */
  733. err_out:
  734. 	SetPageError(page);
  735. 	goto out;
  736. }
  737.  
  738. static void z_erofs_vle_unzip_kickoff(void *ptr, int bios)
  739. {
  740. 	tagptr1_t t = tagptr_init(tagptr1_t, ptr);
  741. 	struct z_erofs_vle_unzip_io *io = tagptr_unfold_ptr(t);
  742. 	bool background = tagptr_unfold_tags(t);
  743.  
  744. 	if (!background) {
  745. 		unsigned long flags;
  746.  
  747. 		spin_lock_irqsave(&io->u.wait.lock, flags);
  748. 		if (!atomic_add_return(bios, &io->pending_bios))
  749. 			wake_up_locked(&io->u.wait);
  750. 		spin_unlock_irqrestore(&io->u.wait.lock, flags);
  751. 		return;
  752. 	}
  753.  
  754. 	if (!atomic_add_return(bios, &io->pending_bios))
  755. 		queue_work(z_erofs_workqueue, &io->u.work);
  756. }
  757.  
  758. static inline void z_erofs_vle_read_endio(struct bio *bio)
  759. {
  760. 	const blk_status_t err = bio->bi_status;
  761. 	unsigned i;
  762. 	struct bio_vec *bvec;
  763. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  764. 	struct address_space *mngda = NULL;
  765. #endif
  766.  
  767. 	bio_for_each_segment_all(bvec, bio, i) {
  768. 		struct page *page = bvec->bv_page;
  769. 		bool cachemngd = false;
  770.  
  771. 		DBG_BUGON(PageUptodate(page));
  772. 		DBG_BUGON(!page->mapping);
  773.  
  774. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  775. 		if (unlikely(mngda == NULL && !z_erofs_is_stagingpage(page))) {
  776. 			struct inode *const inode = page->mapping->host;
  777. 			struct super_block *const sb = inode->i_sb;
  778.  
  779. 			mngda = EROFS_SB(sb)->managed_cache->i_mapping;
  780. 		}
  781.  
  782. 		/*
  783. 		 * If mngda has not gotten, it equals NULL,
  784. 		 * however, page->mapping never be NULL if working properly.
  785. 		 */
  786. 		cachemngd = (page->mapping == mngda);
  787. #endif
  788.  
  789. 		if (unlikely(err))
  790. 			SetPageError(page);
  791. 		else if (cachemngd)
  792. 			SetPageUptodate(page);
  793.  
  794. 		if (cachemngd)
  795. 			unlock_page(page);
  796. 	}
  797.  
  798. 	z_erofs_vle_unzip_kickoff(bio->bi_private, -1);
  799. 	bio_put(bio);
  800. }
  801.  
  802. static struct page *z_pagemap_global[Z_EROFS_VLE_VMAP_GLOBAL_PAGES];
  803. static DEFINE_MUTEX(z_pagemap_global_lock);
  804.  
  805. static int z_erofs_vle_unzip(struct super_block *sb,
  806. 	struct z_erofs_vle_workgroup *grp,
  807. 	struct list_head *page_pool)
  808. {
  809. 	struct erofs_sb_info *const sbi = EROFS_SB(sb);
  810. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  811. 	struct address_space *const mngda = sbi->managed_cache->i_mapping;
  812. #endif
  813. 	const unsigned clusterpages = erofs_clusterpages(sbi);
  814.  
  815. 	struct z_erofs_pagevec_ctor ctor;
  816. 	unsigned int nr_pages;
  817. 	unsigned int sparsemem_pages = 0;
  818. 	struct page *pages_onstack[Z_EROFS_VLE_VMAP_ONSTACK_PAGES];
  819. 	struct page **pages, **compressed_pages, *page;
  820. 	unsigned i, llen;
  821.  
  822. 	enum z_erofs_page_type page_type;
  823. 	bool overlapped;
  824. 	struct z_erofs_vle_work *work;
  825. 	void *vout;
  826. 	int err;
  827.  
  828. 	might_sleep();
  829. 	work = z_erofs_vle_grab_primary_work(grp);
  830. 	DBG_BUGON(!READ_ONCE(work->nr_pages));
  831.  
  832. 	mutex_lock(&work->lock);
  833. 	nr_pages = work->nr_pages;
  834.  
  835. 	if (likely(nr_pages <= Z_EROFS_VLE_VMAP_ONSTACK_PAGES))
  836. 		pages = pages_onstack;
  837. 	else if (nr_pages <= Z_EROFS_VLE_VMAP_GLOBAL_PAGES &&
  838. 		mutex_trylock(&z_pagemap_global_lock))
  839. 		pages = z_pagemap_global;
  840. 	else {
  841. repeat:
  842. 		pages = kvmalloc_array(nr_pages,
  843. 			sizeof(struct page *), GFP_KERNEL);
  844.  
  845. 		/* fallback to global pagemap for the lowmem scenario */
  846. 		if (unlikely(pages == NULL)) {
  847. 			if (nr_pages > Z_EROFS_VLE_VMAP_GLOBAL_PAGES)
  848. 				goto repeat;
  849. 			else {
  850. 				mutex_lock(&z_pagemap_global_lock);
  851. 				pages = z_pagemap_global;
  852. 			}
  853. 		}
  854. 	}
  855.  
  856. 	for (i = 0; i < nr_pages; ++i)
  857. 		pages[i] = NULL;
  858.  
  859. 	err = 0;
  860. 	z_erofs_pagevec_ctor_init(&ctor,
  861. 		Z_EROFS_VLE_INLINE_PAGEVECS, work->pagevec, 0);
  862.  
  863. 	for (i = 0; i < work->vcnt; ++i) {
  864. 		unsigned pagenr;
  865.  
  866. 		page = z_erofs_pagevec_ctor_dequeue(&ctor, &page_type);
  867.  
  868. 		/* all pages in pagevec ought to be valid */
  869. 		DBG_BUGON(page == NULL);
  870. 		DBG_BUGON(page->mapping == NULL);
  871.  
  872. 		if (z_erofs_gather_if_stagingpage(page_pool, page))
  873. 			continue;
  874.  
  875. 		if (page_type == Z_EROFS_VLE_PAGE_TYPE_HEAD)
  876. 			pagenr = 0;
  877. 		else
  878. 			pagenr = z_erofs_onlinepage_index(page);
  879.  
  880. 		DBG_BUGON(pagenr >= nr_pages);
  881.  
  882. 		/*
  883. 		 * currently EROFS doesn't support multiref(dedup),
  884. 		 * so here erroring out one multiref page.
  885. 		 */
  886. 		if (pages[pagenr]) {
  887. 			DBG_BUGON(1);
  888. 			SetPageError(pages[pagenr]);
  889. 			z_erofs_onlinepage_endio(pages[pagenr]);
  890. 			err = -EIO;
  891. 		}
  892. 		pages[pagenr] = page;
  893. 	}
  894. 	sparsemem_pages = i;
  895.  
  896. 	z_erofs_pagevec_ctor_exit(&ctor, true);
  897.  
  898. 	overlapped = false;
  899. 	compressed_pages = grp->compressed_pages;
  900.  
  901. 	for (i = 0; i < clusterpages; ++i) {
  902. 		unsigned pagenr;
  903.  
  904. 		page = compressed_pages[i];
  905.  
  906. 		/* all compressed pages ought to be valid */
  907. 		DBG_BUGON(page == NULL);
  908. 		DBG_BUGON(page->mapping == NULL);
  909.  
  910. 		if (!z_erofs_is_stagingpage(page)) {
  911. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  912. 			if (page->mapping == mngda) {
  913. 				if (unlikely(!PageUptodate(page)))
  914. 					err = -EIO;
  915. 				continue;
  916. 			}
  917. #endif
  918.  
  919. 			/*
  920. 			 * only if non-head page can be selected
  921. 			 * for inplace decompression
  922. 			 */
  923. 			pagenr = z_erofs_onlinepage_index(page);
  924.  
  925. 			DBG_BUGON(pagenr >= nr_pages);
  926. 			if (pages[pagenr]) {
  927. 				DBG_BUGON(1);
  928. 				SetPageError(pages[pagenr]);
  929. 				z_erofs_onlinepage_endio(pages[pagenr]);
  930. 				err = -EIO;
  931. 			}
  932. 			++sparsemem_pages;
  933. 			pages[pagenr] = page;
  934.  
  935. 			overlapped = true;
  936. 		}
  937.  
  938. 		/* PG_error needs checking for inplaced and staging pages */
  939. 		if (unlikely(PageError(page))) {
  940. 			DBG_BUGON(PageUptodate(page));
  941. 			err = -EIO;
  942. 		}
  943. 	}
  944.  
  945. 	if (unlikely(err))
  946. 		goto out;
  947.  
  948. 	llen = (nr_pages << PAGE_SHIFT) - work->pageofs;
  949.  
  950. 	if (z_erofs_vle_workgrp_fmt(grp) == Z_EROFS_VLE_WORKGRP_FMT_PLAIN) {
  951. 		err = z_erofs_vle_plain_copy(compressed_pages, clusterpages,
  952. 			pages, nr_pages, work->pageofs);
  953. 		goto out;
  954. 	}
  955.  
  956. 	if (llen > grp->llen)
  957. 		llen = grp->llen;
  958.  
  959. 	err = z_erofs_vle_unzip_fast_percpu(compressed_pages, clusterpages,
  960. 					    pages, llen, work->pageofs);
  961. 	if (err != -ENOTSUPP)
  962. 		goto out;
  963.  
  964. 	if (sparsemem_pages >= nr_pages)
  965. 		goto skip_allocpage;
  966.  
  967. 	for (i = 0; i < nr_pages; ++i) {
  968. 		if (pages[i] != NULL)
  969. 			continue;
  970.  
  971. 		pages[i] = __stagingpage_alloc(page_pool, GFP_NOFS);
  972. 	}
  973.  
  974. skip_allocpage:
  975. 	vout = erofs_vmap(pages, nr_pages);
  976. 	if (!vout) {
  977. 		err = -ENOMEM;
  978. 		goto out;
  979. 	}
  980.  
  981. 	err = z_erofs_vle_unzip_vmap(compressed_pages,
  982. 		clusterpages, vout, llen, work->pageofs, overlapped);
  983.  
  984. 	erofs_vunmap(vout, nr_pages);
  985.  
  986. out:
  987. 	/* must handle all compressed pages before endding pages */
  988. 	for (i = 0; i < clusterpages; ++i) {
  989. 		page = compressed_pages[i];
  990.  
  991. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  992. 		if (page->mapping == mngda)
  993. 			continue;
  994. #endif
  995. 		/* recycle all individual staging pages */
  996. 		(void)z_erofs_gather_if_stagingpage(page_pool, page);
  997.  
  998. 		WRITE_ONCE(compressed_pages[i], NULL);
  999. 	}
  1000.  
  1001. 	for (i = 0; i < nr_pages; ++i) {
  1002. 		page = pages[i];
  1003. 		if (!page)
  1004. 			continue;
  1005.  
  1006. 		DBG_BUGON(page->mapping == NULL);
  1007.  
  1008. 		/* recycle all individual staging pages */
  1009. 		if (z_erofs_gather_if_stagingpage(page_pool, page))
  1010. 			continue;
  1011.  
  1012. 		if (unlikely(err < 0))
  1013. 			SetPageError(page);
  1014.  
  1015. 		z_erofs_onlinepage_endio(page);
  1016. 	}
  1017.  
  1018. 	if (pages == z_pagemap_global)
  1019. 		mutex_unlock(&z_pagemap_global_lock);
  1020. 	else if (unlikely(pages != pages_onstack))
  1021. 		kvfree(pages);
  1022.  
  1023. 	work->nr_pages = 0;
  1024. 	work->vcnt = 0;
  1025.  
  1026. 	/* all work locks MUST be taken before the following line */
  1027.  
  1028. 	WRITE_ONCE(grp->next, Z_EROFS_VLE_WORKGRP_NIL);
  1029.  
  1030. 	/* all work locks SHOULD be released right now */
  1031. 	mutex_unlock(&work->lock);
  1032.  
  1033. 	z_erofs_vle_work_release(work);
  1034. 	return err;
  1035. }
  1036.  
  1037. static void z_erofs_vle_unzip_all(struct super_block *sb,
  1038. 				  struct z_erofs_vle_unzip_io *io,
  1039. 				  struct list_head *page_pool)
  1040. {
  1041. 	z_erofs_vle_owned_workgrp_t owned = io->head;
  1042.  
  1043. 	while (owned != Z_EROFS_VLE_WORKGRP_TAIL_CLOSED) {
  1044. 		struct z_erofs_vle_workgroup *grp;
  1045.  
  1046. 		/* no possible that 'owned' equals Z_EROFS_WORK_TPTR_TAIL */
  1047. 		DBG_BUGON(owned == Z_EROFS_VLE_WORKGRP_TAIL);
  1048.  
  1049. 		/* no possible that 'owned' equals NULL */
  1050. 		DBG_BUGON(owned == Z_EROFS_VLE_WORKGRP_NIL);
  1051.  
  1052. 		grp = owned;
  1053. 		owned = READ_ONCE(grp->next);
  1054.  
  1055. 		z_erofs_vle_unzip(sb, grp, page_pool);
  1056. 	}
  1057. }
  1058.  
  1059. static void z_erofs_vle_unzip_wq(struct work_struct *work)
  1060. {
  1061. 	struct z_erofs_vle_unzip_io_sb *iosb = container_of(work,
  1062. 		struct z_erofs_vle_unzip_io_sb, io.u.work);
  1063. 	LIST_HEAD(page_pool);
  1064.  
  1065. 	DBG_BUGON(iosb->io.head == Z_EROFS_VLE_WORKGRP_TAIL_CLOSED);
  1066. 	z_erofs_vle_unzip_all(iosb->sb, &iosb->io, &page_pool);
  1067.  
  1068. 	put_pages_list(&page_pool);
  1069. 	kvfree(iosb);
  1070. }
  1071.  
  1072. static inline struct z_erofs_vle_unzip_io *
  1073. prepare_io_handler(struct super_block *sb,
  1074. 		   struct z_erofs_vle_unzip_io *io,
  1075. 		   bool background)
  1076. {
  1077. 	struct z_erofs_vle_unzip_io_sb *iosb;
  1078.  
  1079. 	if (!background) {
  1080. 		/* waitqueue available for foreground io */
  1081. 		BUG_ON(io == NULL);
  1082.  
  1083. 		init_waitqueue_head(&io->u.wait);
  1084. 		atomic_set(&io->pending_bios, 0);
  1085. 		goto out;
  1086. 	}
  1087.  
  1088. 	if (io != NULL)
  1089. 		BUG();
  1090. 	else {
  1091. 		/* allocate extra io descriptor for background io */
  1092. 		iosb = kvzalloc(sizeof(struct z_erofs_vle_unzip_io_sb),
  1093. 			GFP_KERNEL | __GFP_NOFAIL);
  1094. 		BUG_ON(iosb == NULL);
  1095.  
  1096. 		io = &iosb->io;
  1097. 	}
  1098.  
  1099. 	iosb->sb = sb;
  1100. 	INIT_WORK(&io->u.work, z_erofs_vle_unzip_wq);
  1101. out:
  1102. 	io->head = Z_EROFS_VLE_WORKGRP_TAIL_CLOSED;
  1103. 	return io;
  1104. }
  1105.  
  1106. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  1107. /* true - unlocked (noio), false - locked (need submit io) */
  1108. static inline bool recover_managed_page(struct z_erofs_vle_workgroup *grp,
  1109. 					struct page *page)
  1110. {
  1111. 	wait_on_page_locked(page);
  1112. 	if (PagePrivate(page) && PageUptodate(page))
  1113. 		return true;
  1114.  
  1115. 	lock_page(page);
  1116. 	ClearPageError(page);
  1117.  
  1118. 	if (unlikely(!PagePrivate(page))) {
  1119. 		set_page_private(page, (unsigned long)grp);
  1120. 		SetPagePrivate(page);
  1121. 	}
  1122. 	if (unlikely(PageUptodate(page))) {
  1123. 		unlock_page(page);
  1124. 		return true;
  1125. 	}
  1126. 	return false;
  1127. }
  1128.  
  1129. #define __FSIO_1 1
  1130. #else
  1131. #define __FSIO_1 0
  1132. #endif
  1133.  
  1134. static bool z_erofs_vle_submit_all(struct super_block *sb,
  1135. 				   z_erofs_vle_owned_workgrp_t owned_head,
  1136. 				   struct list_head *pagepool,
  1137. 				   struct z_erofs_vle_unzip_io *fg_io,
  1138. 				   bool force_fg)
  1139. {
  1140. 	struct erofs_sb_info *const sbi = EROFS_SB(sb);
  1141. 	const unsigned clusterpages = erofs_clusterpages(sbi);
  1142. 	const gfp_t gfp = GFP_NOFS;
  1143. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  1144. 	struct address_space *const mngda = sbi->managed_cache->i_mapping;
  1145. 	struct z_erofs_vle_workgroup *lstgrp_noio = NULL, *lstgrp_io = NULL;
  1146. #endif
  1147. 	struct z_erofs_vle_unzip_io *ios[1 + __FSIO_1];
  1148. 	struct bio *bio;
  1149. 	tagptr1_t bi_private;
  1150. 	/* since bio will be NULL, no need to initialize last_index */
  1151. 	pgoff_t uninitialized_var(last_index);
  1152. 	bool force_submit = false;
  1153. 	unsigned nr_bios;
  1154.  
  1155. 	if (unlikely(owned_head == Z_EROFS_VLE_WORKGRP_TAIL))
  1156. 		return false;
  1157.  
  1158. 	/*
  1159. 	 * force_fg == 1, (io, fg_io[0]) no io, (io, fg_io[1]) need submit io
  1160. 	 * force_fg == 0, (io, fg_io[0]) no io; (io[1], bg_io) need submit io
  1161. 	 */
  1162. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  1163. 	ios[0] = prepare_io_handler(sb, fg_io + 0, false);
  1164. #endif
  1165.  
  1166. 	if (force_fg) {
  1167. 		ios[__FSIO_1] = prepare_io_handler(sb, fg_io + __FSIO_1, false);
  1168. 		bi_private = tagptr_fold(tagptr1_t, ios[__FSIO_1], 0);
  1169. 	} else {
  1170. 		ios[__FSIO_1] = prepare_io_handler(sb, NULL, true);
  1171. 		bi_private = tagptr_fold(tagptr1_t, ios[__FSIO_1], 1);
  1172. 	}
  1173.  
  1174. 	nr_bios = 0;
  1175. 	force_submit = false;
  1176. 	bio = NULL;
  1177.  
  1178. 	/* by default, all need io submission */
  1179. 	ios[__FSIO_1]->head = owned_head;
  1180.  
  1181. 	do {
  1182. 		struct z_erofs_vle_workgroup *grp;
  1183. 		struct page **compressed_pages, *oldpage, *page;
  1184. 		pgoff_t first_index;
  1185. 		unsigned i = 0;
  1186. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  1187. 		unsigned int noio = 0;
  1188. 		bool cachemngd;
  1189. #endif
  1190. 		int err;
  1191.  
  1192. 		/* no possible 'owned_head' equals the following */
  1193. 		DBG_BUGON(owned_head == Z_EROFS_VLE_WORKGRP_TAIL_CLOSED);
  1194. 		DBG_BUGON(owned_head == Z_EROFS_VLE_WORKGRP_NIL);
  1195.  
  1196. 		grp = owned_head;
  1197.  
  1198. 		/* close the main owned chain at first */
  1199. 		owned_head = cmpxchg(&grp->next, Z_EROFS_VLE_WORKGRP_TAIL,
  1200. 			Z_EROFS_VLE_WORKGRP_TAIL_CLOSED);
  1201.  
  1202. 		first_index = grp->obj.index;
  1203. 		compressed_pages = grp->compressed_pages;
  1204.  
  1205. 		force_submit |= (first_index != last_index + 1);
  1206. repeat:
  1207. 		/* fulfill all compressed pages */
  1208. 		oldpage = page = READ_ONCE(compressed_pages[i]);
  1209.  
  1210. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  1211. 		cachemngd = false;
  1212.  
  1213. 		if (page == EROFS_UNALLOCATED_CACHED_PAGE) {
  1214. 			cachemngd = true;
  1215. 			goto do_allocpage;
  1216. 		} else if (page != NULL) {
  1217. 			if (page->mapping != mngda)
  1218. 				BUG_ON(PageUptodate(page));
  1219. 			else if (recover_managed_page(grp, page)) {
  1220. 				/* page is uptodate, skip io submission */
  1221. 				force_submit = true;
  1222. 				++noio;
  1223. 				goto skippage;
  1224. 			}
  1225. 		} else {
  1226. do_allocpage:
  1227. #else
  1228. 		if (page != NULL)
  1229. 			BUG_ON(PageUptodate(page));
  1230. 		else {
  1231. #endif
  1232. 			page = __stagingpage_alloc(pagepool, gfp);
  1233.  
  1234. 			if (oldpage != cmpxchg(compressed_pages + i,
  1235. 				oldpage, page)) {
  1236. 				list_add(&page->lru, pagepool);
  1237. 				goto repeat;
  1238. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  1239. 			} else if (cachemngd && !add_to_page_cache_lru(page,
  1240. 				mngda, first_index + i, gfp)) {
  1241. 				set_page_private(page, (unsigned long)grp);
  1242. 				SetPagePrivate(page);
  1243. #endif
  1244. 			}
  1245. 		}
  1246.  
  1247. 		if (bio != NULL && force_submit) {
  1248. submit_bio_retry:
  1249. 			__submit_bio(bio, REQ_OP_READ, 0);
  1250. 			bio = NULL;
  1251. 		}
  1252.  
  1253. 		if (bio == NULL) {
  1254. 			bio = prepare_bio(sb, first_index + i,
  1255. 				BIO_MAX_PAGES, z_erofs_vle_read_endio);
  1256. 			bio->bi_private = tagptr_cast_ptr(bi_private);
  1257.  
  1258. 			++nr_bios;
  1259. 		}
  1260.  
  1261. 		err = bio_add_page(bio, page, PAGE_SIZE, 0);
  1262. 		if (err < PAGE_SIZE)
  1263. 			goto submit_bio_retry;
  1264.  
  1265. 		force_submit = false;
  1266. 		last_index = first_index + i;
  1267. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  1268. skippage:
  1269. #endif
  1270. 		if (++i < clusterpages)
  1271. 			goto repeat;
  1272.  
  1273. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  1274. 		if (noio < clusterpages) {
  1275. 			lstgrp_io = grp;
  1276. 		} else {
  1277. 			z_erofs_vle_owned_workgrp_t iogrp_next =
  1278. 				owned_head == Z_EROFS_VLE_WORKGRP_TAIL ?
  1279. 				Z_EROFS_VLE_WORKGRP_TAIL_CLOSED :
  1280. 				owned_head;
  1281.  
  1282. 			if (lstgrp_io == NULL)
  1283. 				ios[1]->head = iogrp_next;
  1284. 			else
  1285. 				WRITE_ONCE(lstgrp_io->next, iogrp_next);
  1286.  
  1287. 			if (lstgrp_noio == NULL)
  1288. 				ios[0]->head = grp;
  1289. 			else
  1290. 				WRITE_ONCE(lstgrp_noio->next, grp);
  1291.  
  1292. 			lstgrp_noio = grp;
  1293. 		}
  1294. #endif
  1295. 	} while (owned_head != Z_EROFS_VLE_WORKGRP_TAIL);
  1296.  
  1297. 	if (bio != NULL)
  1298. 		__submit_bio(bio, REQ_OP_READ, 0);
  1299.  
  1300. #ifndef EROFS_FS_HAS_MANAGED_CACHE
  1301. 	BUG_ON(!nr_bios);
  1302. #else
  1303. 	if (lstgrp_noio != NULL)
  1304. 		WRITE_ONCE(lstgrp_noio->next, Z_EROFS_VLE_WORKGRP_TAIL_CLOSED);
  1305.  
  1306. 	if (!force_fg && !nr_bios) {
  1307. 		kvfree(container_of(ios[1],
  1308. 			struct z_erofs_vle_unzip_io_sb, io));
  1309. 		return true;
  1310. 	}
  1311. #endif
  1312.  
  1313. 	z_erofs_vle_unzip_kickoff(tagptr_cast_ptr(bi_private), nr_bios);
  1314. 	return true;
  1315. }
  1316.  
  1317. static void z_erofs_submit_and_unzip(struct z_erofs_vle_frontend *f,
  1318. 				     struct list_head *pagepool,
  1319. 				     bool force_fg)
  1320. {
  1321. 	struct super_block *sb = f->inode->i_sb;
  1322. 	struct z_erofs_vle_unzip_io io[1 + __FSIO_1];
  1323.  
  1324. 	if (!z_erofs_vle_submit_all(sb, f->owned_head, pagepool, io, force_fg))
  1325. 		return;
  1326.  
  1327. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  1328. 	z_erofs_vle_unzip_all(sb, &io[0], pagepool);
  1329. #endif
  1330. 	if (!force_fg)
  1331. 		return;
  1332.  
  1333. 	/* wait until all bios are completed */
  1334. 	wait_event(io[__FSIO_1].u.wait,
  1335. 		!atomic_read(&io[__FSIO_1].pending_bios));
  1336.  
  1337. 	/* let's synchronous decompression */
  1338. 	z_erofs_vle_unzip_all(sb, &io[__FSIO_1], pagepool);
  1339. }
  1340.  
  1341. static int z_erofs_vle_normalaccess_readpage(struct file *file,
  1342. 					     struct page *page)
  1343. {
  1344. 	struct inode *const inode = page->mapping->host;
  1345. 	struct z_erofs_vle_frontend f = VLE_FRONTEND_INIT(inode);
  1346. 	int err;
  1347. 	LIST_HEAD(pagepool);
  1348.  
  1349. #if (EROFS_FS_ZIP_CACHE_LVL >= 2)
  1350. 	f.cachedzone_la = page->index << PAGE_SHIFT;
  1351. #endif
  1352. 	err = z_erofs_do_read_page(&f, page, &pagepool);
  1353. 	(void)z_erofs_vle_work_iter_end(&f.builder);
  1354.  
  1355. 	/* if some compressed cluster ready, need submit them anyway */
  1356. 	z_erofs_submit_and_unzip(&f, &pagepool, true);
  1357.  
  1358. 	if (err)
  1359. 		errln("%s, failed to read, err [%d]", __func__, err);
  1360.  
  1361. 	if (f.m_iter.mpage != NULL)
  1362. 		put_page(f.m_iter.mpage);
  1363.  
  1364. 	/* clean up the remaining free pages */
  1365. 	put_pages_list(&pagepool);
  1366. 	return err;
  1367. }
  1368.  
  1369. static inline int __z_erofs_vle_normalaccess_readpages(
  1370. 	struct file *filp,
  1371. 	struct address_space *mapping,
  1372. 	struct list_head *pages, unsigned nr_pages, bool sync)
  1373. {
  1374. 	struct inode *const inode = mapping->host;
  1375.  
  1376. 	struct z_erofs_vle_frontend f = VLE_FRONTEND_INIT(inode);
  1377. 	gfp_t gfp = mapping_gfp_constraint(mapping, GFP_KERNEL);
  1378. 	struct page *head = NULL;
  1379. 	LIST_HEAD(pagepool);
  1380.  
  1381. #if (EROFS_FS_ZIP_CACHE_LVL >= 2)
  1382. 	f.cachedzone_la = lru_to_page(pages)->index << PAGE_SHIFT;
  1383. #endif
  1384. 	for (; nr_pages; --nr_pages) {
  1385. 		struct page *page = lru_to_page(pages);
  1386.  
  1387. 		prefetchw(&page->flags);
  1388. 		list_del(&page->lru);
  1389.  
  1390. 		if (add_to_page_cache_lru(page, mapping, page->index, gfp)) {
  1391. 			list_add(&page->lru, &pagepool);
  1392. 			continue;
  1393. 		}
  1394.  
  1395. 		set_page_private(page, (unsigned long)head);
  1396. 		head = page;
  1397. 	}
  1398.  
  1399. 	while (head != NULL) {
  1400. 		struct page *page = head;
  1401. 		int err;
  1402.  
  1403. 		/* traversal in reverse order */
  1404. 		head = (void *)page_private(page);
  1405.  
  1406. 		err = z_erofs_do_read_page(&f, page, &pagepool);
  1407. 		if (err) {
  1408. 			struct erofs_vnode *vi = EROFS_V(inode);
  1409.  
  1410. 			errln("%s, readahead error at page %lu of nid %llu",
  1411. 				__func__, page->index, vi->nid);
  1412. 		}
  1413.  
  1414. 		put_page(page);
  1415. 	}
  1416.  
  1417. 	(void)z_erofs_vle_work_iter_end(&f.builder);
  1418.  
  1419. 	z_erofs_submit_and_unzip(&f, &pagepool, sync);
  1420.  
  1421. 	if (f.m_iter.mpage != NULL)
  1422. 		put_page(f.m_iter.mpage);
  1423.  
  1424. 	/* clean up the remaining free pages */
  1425. 	put_pages_list(&pagepool);
  1426. 	return 0;
  1427. }
  1428.  
  1429. static int z_erofs_vle_normalaccess_readpages(
  1430. 	struct file *filp,
  1431. 	struct address_space *mapping,
  1432. 	struct list_head *pages, unsigned nr_pages)
  1433. {
  1434. 	return __z_erofs_vle_normalaccess_readpages(filp,
  1435. 		mapping, pages, nr_pages,
  1436. 		nr_pages < 4 /* sync */);
  1437. }
  1438.  
  1439. const struct address_space_operations z_erofs_vle_normalaccess_aops = {
  1440. 	.readpage = z_erofs_vle_normalaccess_readpage,
  1441. 	.readpages = z_erofs_vle_normalaccess_readpages,
  1442. };
  1443.  
  1444. #define __vle_cluster_advise(x, bit, bits) \
  1445. 	((le16_to_cpu(x) >> (bit)) & ((1 << (bits)) - 1))
  1446.  
  1447. #define __vle_cluster_type(advise) __vle_cluster_advise(advise, \
  1448. 	Z_EROFS_VLE_DI_CLUSTER_TYPE_BIT, Z_EROFS_VLE_DI_CLUSTER_TYPE_BITS)
  1449.  
  1450. enum {
  1451. 	Z_EROFS_VLE_CLUSTER_TYPE_PLAIN,
  1452. 	Z_EROFS_VLE_CLUSTER_TYPE_HEAD,
  1453. 	Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD,
  1454. 	Z_EROFS_VLE_CLUSTER_TYPE_RESERVED,
  1455. 	Z_EROFS_VLE_CLUSTER_TYPE_MAX
  1456. };
  1457.  
  1458. #define vle_cluster_type(di)	\
  1459. 	__vle_cluster_type((di)->di_advise)
  1460.  
  1461. static inline unsigned
  1462. vle_compressed_index_clusterofs(unsigned clustersize,
  1463. 	struct z_erofs_vle_decompressed_index *di)
  1464. {
  1465. 	debugln("%s, vle=%pK, advise=%x (type %u), clusterofs=%x blkaddr=%x",
  1466. 		__func__, di, di->di_advise, vle_cluster_type(di),
  1467. 		di->di_clusterofs, di->di_u.blkaddr);
  1468.  
  1469. 	switch (vle_cluster_type(di)) {
  1470. 	case Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD:
  1471. 		break;
  1472. 	case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:
  1473. 	case Z_EROFS_VLE_CLUSTER_TYPE_HEAD:
  1474. 		return di->di_clusterofs;
  1475. 	default:
  1476. 		BUG_ON(1);
  1477. 	}
  1478. 	return clustersize;
  1479. }
  1480.  
  1481. static inline erofs_blk_t
  1482. vle_extent_blkaddr(struct inode *inode, pgoff_t index)
  1483. {
  1484. 	struct erofs_sb_info *sbi = EROFS_I_SB(inode);
  1485. 	struct erofs_vnode *vi = EROFS_V(inode);
  1486.  
  1487. 	unsigned ofs = Z_EROFS_VLE_EXTENT_ALIGN(vi->inode_isize +
  1488. 		vi->xattr_isize) + sizeof(struct erofs_extent_header) +
  1489. 		index * sizeof(struct z_erofs_vle_decompressed_index);
  1490.  
  1491. 	return erofs_blknr(iloc(sbi, vi->nid) + ofs);
  1492. }
  1493.  
  1494. static inline unsigned int
  1495. vle_extent_blkoff(struct inode *inode, pgoff_t index)
  1496. {
  1497. 	struct erofs_sb_info *sbi = EROFS_I_SB(inode);
  1498. 	struct erofs_vnode *vi = EROFS_V(inode);
  1499.  
  1500. 	unsigned ofs = Z_EROFS_VLE_EXTENT_ALIGN(vi->inode_isize +
  1501. 		vi->xattr_isize) + sizeof(struct erofs_extent_header) +
  1502. 		index * sizeof(struct z_erofs_vle_decompressed_index);
  1503.  
  1504. 	return erofs_blkoff(iloc(sbi, vi->nid) + ofs);
  1505. }
  1506.  
  1507. /*
  1508.  * Variable-sized Logical Extent (Fixed Physical Cluster) Compression Mode
  1509.  * ---
  1510.  * VLE compression mode attempts to compress a number of logical data into
  1511.  * a physical cluster with a fixed size.
  1512.  * VLE compression mode uses "struct z_erofs_vle_decompressed_index".
  1513.  */
  1514. static erofs_off_t vle_get_logical_extent_head(
  1515. 	struct inode *inode,
  1516. 	struct page **page_iter,
  1517. 	void **kaddr_iter,
  1518. 	unsigned lcn,	/* logical cluster number */
  1519. 	erofs_blk_t *pcn,
  1520. 	unsigned *flags)
  1521. {
  1522. 	/* for extent meta */
  1523. 	struct page *page = *page_iter;
  1524. 	erofs_blk_t blkaddr = vle_extent_blkaddr(inode, lcn);
  1525. 	struct z_erofs_vle_decompressed_index *di;
  1526. 	unsigned long long ofs;
  1527. 	const unsigned int clusterbits = EROFS_SB(inode->i_sb)->clusterbits;
  1528. 	const unsigned int clustersize = 1 << clusterbits;
  1529. 	unsigned int delta0;
  1530.  
  1531. 	if (page->index != blkaddr) {
  1532. 		kunmap_atomic(*kaddr_iter);
  1533. 		unlock_page(page);
  1534. 		put_page(page);
  1535.  
  1536. 		*page_iter = page = erofs_get_meta_page(inode->i_sb,
  1537. 			blkaddr, false);
  1538. 		*kaddr_iter = kmap_atomic(page);
  1539. 	}
  1540.  
  1541. 	di = *kaddr_iter + vle_extent_blkoff(inode, lcn);
  1542. 	switch (vle_cluster_type(di)) {
  1543. 	case Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD:
  1544. 		delta0 = le16_to_cpu(di->di_u.delta[0]);
  1545. 		DBG_BUGON(!delta0);
  1546. 		DBG_BUGON(lcn < delta0);
  1547.  
  1548. 		ofs = vle_get_logical_extent_head(inode,
  1549. 			page_iter, kaddr_iter,
  1550. 			lcn - delta0, pcn, flags);
  1551. 		break;
  1552. 	case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:
  1553. 		*flags ^= EROFS_MAP_ZIPPED;
  1554. 	case Z_EROFS_VLE_CLUSTER_TYPE_HEAD:
  1555. 		/* clustersize should be a power of two */
  1556. 		ofs = ((unsigned long long)lcn << clusterbits) +
  1557. 			(le16_to_cpu(di->di_clusterofs) & (clustersize - 1));
  1558. 		*pcn = le32_to_cpu(di->di_u.blkaddr);
  1559. 		break;
  1560. 	default:
  1561. 		BUG_ON(1);
  1562. 	}
  1563. 	return ofs;
  1564. }
  1565.  
  1566. int z_erofs_map_blocks_iter(struct inode *inode,
  1567. 	struct erofs_map_blocks *map,
  1568. 	struct page **mpage_ret, int flags)
  1569. {
  1570. 	/* logicial extent (start, end) offset */
  1571. 	unsigned long long ofs, end;
  1572. 	struct z_erofs_vle_decompressed_index *di;
  1573. 	erofs_blk_t e_blkaddr, pcn;
  1574. 	unsigned lcn, logical_cluster_ofs, cluster_type;
  1575. 	u32 ofs_rem;
  1576. 	struct page *mpage = *mpage_ret;
  1577. 	void *kaddr;
  1578. 	bool initial;
  1579. 	const unsigned int clusterbits = EROFS_SB(inode->i_sb)->clusterbits;
  1580. 	const unsigned int clustersize = 1 << clusterbits;
  1581. 	int err = 0;
  1582.  
  1583. 	/* if both m_(l,p)len are 0, regularize l_lblk, l_lofs, etc... */
  1584. 	initial = !map->m_llen;
  1585.  
  1586. 	/* when trying to read beyond EOF, leave it unmapped */
  1587. 	if (unlikely(map->m_la >= inode->i_size)) {
  1588. 		BUG_ON(!initial);
  1589. 		map->m_llen = map->m_la + 1 - inode->i_size;
  1590. 		map->m_la = inode->i_size - 1;
  1591. 		map->m_flags = 0;
  1592. 		goto out;
  1593. 	}
  1594.  
  1595. 	debugln("%s, m_la %llu m_llen %llu --- start", __func__,
  1596. 		map->m_la, map->m_llen);
  1597.  
  1598. 	ofs = map->m_la + map->m_llen;
  1599.  
  1600. 	/* clustersize should be power of two */
  1601. 	lcn = ofs >> clusterbits;
  1602. 	ofs_rem = ofs & (clustersize - 1);
  1603.  
  1604. 	e_blkaddr = vle_extent_blkaddr(inode, lcn);
  1605.  
  1606. 	if (mpage == NULL || mpage->index != e_blkaddr) {
  1607. 		if (mpage != NULL)
  1608. 			put_page(mpage);
  1609.  
  1610. 		mpage = erofs_get_meta_page(inode->i_sb, e_blkaddr, false);
  1611. 		*mpage_ret = mpage;
  1612. 	} else {
  1613. 		lock_page(mpage);
  1614. 		DBG_BUGON(!PageUptodate(mpage));
  1615. 	}
  1616.  
  1617. 	kaddr = kmap_atomic(mpage);
  1618. 	di = kaddr + vle_extent_blkoff(inode, lcn);
  1619.  
  1620. 	debugln("%s, lcn %u e_blkaddr %u e_blkoff %u", __func__, lcn,
  1621. 		e_blkaddr, vle_extent_blkoff(inode, lcn));
  1622.  
  1623. 	logical_cluster_ofs = vle_compressed_index_clusterofs(clustersize, di);
  1624. 	if (!initial) {
  1625. 		/* [walking mode] 'map' has been already initialized */
  1626. 		map->m_llen += logical_cluster_ofs;
  1627. 		goto unmap_out;
  1628. 	}
  1629.  
  1630. 	/* by default, compressed */
  1631. 	map->m_flags |= EROFS_MAP_ZIPPED;
  1632.  
  1633. 	end = (u64)(lcn + 1) * clustersize;
  1634.  
  1635. 	cluster_type = vle_cluster_type(di);
  1636.  
  1637. 	switch (cluster_type) {
  1638. 	case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:
  1639. 		if (ofs_rem >= logical_cluster_ofs)
  1640. 			map->m_flags ^= EROFS_MAP_ZIPPED;
  1641. 		/* fallthrough */
  1642. 	case Z_EROFS_VLE_CLUSTER_TYPE_HEAD:
  1643. 		if (ofs_rem == logical_cluster_ofs) {
  1644. 			pcn = le32_to_cpu(di->di_u.blkaddr);
  1645. 			goto exact_hitted;
  1646. 		}
  1647.  
  1648. 		if (ofs_rem > logical_cluster_ofs) {
  1649. 			ofs = lcn * clustersize | logical_cluster_ofs;
  1650. 			pcn = le32_to_cpu(di->di_u.blkaddr);
  1651. 			break;
  1652. 		}
  1653.  
  1654. 		/* logical cluster number should be >= 1 */
  1655. 		if (unlikely(!lcn)) {
  1656. 			errln("invalid logical cluster 0 at nid %llu",
  1657. 				EROFS_V(inode)->nid);
  1658. 			err = -EIO;
  1659. 			goto unmap_out;
  1660. 		}
  1661. 		end = (lcn-- * clustersize) | logical_cluster_ofs;
  1662. 		/* fallthrough */
  1663. 	case Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD:
  1664. 		/* get the correspoinding first chunk */
  1665. 		ofs = vle_get_logical_extent_head(inode, mpage_ret,
  1666. 			&kaddr, lcn, &pcn, &map->m_flags);
  1667. 		mpage = *mpage_ret;
  1668. 		break;
  1669. 	default:
  1670. 		errln("unknown cluster type %u at offset %llu of nid %llu",
  1671. 			cluster_type, ofs, EROFS_V(inode)->nid);
  1672. 		err = -EIO;
  1673. 		goto unmap_out;
  1674. 	}
  1675.  
  1676. 	map->m_la = ofs;
  1677. exact_hitted:
  1678. 	map->m_llen = end - ofs;
  1679. 	map->m_plen = clustersize;
  1680. 	map->m_pa = blknr_to_addr(pcn);
  1681. 	map->m_flags |= EROFS_MAP_MAPPED;
  1682. unmap_out:
  1683. 	kunmap_atomic(kaddr);
  1684. 	unlock_page(mpage);
  1685. out:
  1686. 	debugln("%s, m_la %llu m_pa %llu m_llen %llu m_plen %llu m_flags 0%o",
  1687. 		__func__, map->m_la, map->m_pa,
  1688. 		map->m_llen, map->m_plen, map->m_flags);
  1689.  
  1690. 	/* aggressively BUG_ON iff CONFIG_EROFS_FS_DEBUG is on */
  1691. 	DBG_BUGON(err < 0);
  1692. 	return err;
  1693. }
  1694.

unzip_vle_lz4.c

  1. // SPDX-License-Identifier: GPL-2.0
  2. /*
  3.  * linux/drivers/staging/erofs/unzip_vle_lz4.c
  4.  *
  5.  * Copyright (C) 2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #include "unzip_vle.h"
  14.  
  15. #if Z_EROFS_CLUSTER_MAX_PAGES > Z_EROFS_VLE_INLINE_PAGEVECS
  16. #define EROFS_PERCPU_NR_PAGES   Z_EROFS_CLUSTER_MAX_PAGES
  17. #else
  18. #define EROFS_PERCPU_NR_PAGES   Z_EROFS_VLE_INLINE_PAGEVECS
  19. #endif
  20.  
  21. static struct {
  22. 	char data[PAGE_SIZE * EROFS_PERCPU_NR_PAGES];
  23. } erofs_pcpubuf[NR_CPUS];
  24.  
  25. int z_erofs_vle_plain_copy(struct page **compressed_pages,
  26. 			   unsigned clusterpages,
  27. 			   struct page **pages,
  28. 			   unsigned nr_pages,
  29. 			   unsigned short pageofs)
  30. {
  31. 	unsigned i, j;
  32. 	void *src = NULL;
  33. 	const unsigned righthalf = PAGE_SIZE - pageofs;
  34. 	char *percpu_data;
  35. 	bool mirrored[Z_EROFS_CLUSTER_MAX_PAGES] = { 0 };
  36.  
  37. 	preempt_disable();
  38. 	percpu_data = erofs_pcpubuf[smp_processor_id()].data;
  39.  
  40. 	j = 0;
  41. 	for (i = 0; i < nr_pages; j = i++) {
  42. 		struct page *page = pages[i];
  43. 		void *dst;
  44.  
  45. 		if (page == NULL) {
  46. 			if (src != NULL) {
  47. 				if (!mirrored[j])
  48. 					kunmap_atomic(src);
  49. 				src = NULL;
  50. 			}
  51. 			continue;
  52. 		}
  53.  
  54. 		dst = kmap_atomic(page);
  55.  
  56. 		for (; j < clusterpages; ++j) {
  57. 			if (compressed_pages[j] != page)
  58. 				continue;
  59.  
  60. 			DBG_BUGON(mirrored[j]);
  61. 			memcpy(percpu_data + j * PAGE_SIZE, dst, PAGE_SIZE);
  62. 			mirrored[j] = true;
  63. 			break;
  64. 		}
  65.  
  66. 		if (i) {
  67. 			if (src == NULL)
  68. 				src = mirrored[i-1] ?
  69. 					percpu_data + (i-1) * PAGE_SIZE :
  70. 					kmap_atomic(compressed_pages[i-1]);
  71.  
  72. 			memcpy(dst, src + righthalf, pageofs);
  73.  
  74. 			if (!mirrored[i-1])
  75. 				kunmap_atomic(src);
  76.  
  77. 			if (unlikely(i >= clusterpages)) {
  78. 				kunmap_atomic(dst);
  79. 				break;
  80. 			}
  81. 		}
  82.  
  83. 		if (!righthalf)
  84. 			src = NULL;
  85. 		else {
  86. 			src = mirrored[i] ? percpu_data + i * PAGE_SIZE :
  87. 				kmap_atomic(compressed_pages[i]);
  88.  
  89. 			memcpy(dst + pageofs, src, righthalf);
  90. 		}
  91.  
  92. 		kunmap_atomic(dst);
  93. 	}
  94.  
  95. 	if (src != NULL && !mirrored[j])
  96. 		kunmap_atomic(src);
  97.  
  98. 	preempt_enable();
  99. 	return 0;
  100. }
  101.  
  102. extern int z_erofs_unzip_lz4(void *in, void *out, size_t inlen, size_t outlen);
  103.  
  104. int z_erofs_vle_unzip_fast_percpu(struct page **compressed_pages,
  105. 				  unsigned clusterpages,
  106. 				  struct page **pages,
  107. 				  unsigned outlen,
  108. 				  unsigned short pageofs)
  109. {
  110. 	void *vin, *vout;
  111. 	unsigned nr_pages, i, j;
  112. 	int ret;
  113.  
  114. 	if (outlen + pageofs > EROFS_PERCPU_NR_PAGES * PAGE_SIZE)
  115. 		return -ENOTSUPP;
  116.  
  117. 	nr_pages = DIV_ROUND_UP(outlen + pageofs, PAGE_SIZE);
  118.  
  119. 	if (clusterpages == 1) {
  120. 		vin = kmap_atomic(compressed_pages[0]);
  121. 	} else {
  122. 		vin = erofs_vmap(compressed_pages, clusterpages);
  123. 		if (!vin)
  124. 			return -ENOMEM;
  125. 	}
  126.  
  127. 	preempt_disable();
  128. 	vout = erofs_pcpubuf[smp_processor_id()].data;
  129.  
  130. 	ret = z_erofs_unzip_lz4(vin, vout + pageofs,
  131. 		clusterpages * PAGE_SIZE, outlen);
  132.  
  133. 	if (ret < 0)
  134. 		goto out;
  135. 	ret = 0;
  136.  
  137. 	for (i = 0; i < nr_pages; ++i) {
  138. 		j = min((unsigned)PAGE_SIZE - pageofs, outlen);
  139.  
  140. 		if (pages[i] != NULL) {
  141. 			if (clusterpages == 1 &&
  142. 			    pages[i] == compressed_pages[0]) {
  143. 				memcpy(vin + pageofs, vout + pageofs, j);
  144. 			} else {
  145. 				void *dst = kmap_atomic(pages[i]);
  146.  
  147. 				memcpy(dst + pageofs, vout + pageofs, j);
  148. 				kunmap_atomic(dst);
  149. 			}
  150. 		}
  151. 		vout += PAGE_SIZE;
  152. 		outlen -= j;
  153. 		pageofs = 0;
  154. 	}
  155.  
  156. out:
  157. 	preempt_enable();
  158.  
  159. 	if (clusterpages == 1)
  160. 		kunmap_atomic(vin);
  161. 	else
  162. 		erofs_vunmap(vin, clusterpages);
  163.  
  164. 	return ret;
  165. }
  166.  
  167. int z_erofs_vle_unzip_vmap(struct page **compressed_pages,
  168. 			   unsigned clusterpages,
  169. 			   void *vout,
  170. 			   unsigned llen,
  171. 			   unsigned short pageofs,
  172. 			   bool overlapped)
  173. {
  174. 	void *vin;
  175. 	unsigned i;
  176. 	int ret;
  177.  
  178. 	if (overlapped) {
  179. 		preempt_disable();
  180. 		vin = erofs_pcpubuf[smp_processor_id()].data;
  181.  
  182. 		for (i = 0; i < clusterpages; ++i) {
  183. 			void *t = kmap_atomic(compressed_pages[i]);
  184.  
  185. 			memcpy(vin + PAGE_SIZE *i, t, PAGE_SIZE);
  186. 			kunmap_atomic(t);
  187. 		}
  188. 	} else if (clusterpages == 1)
  189. 		vin = kmap_atomic(compressed_pages[0]);
  190. 	else {
  191. 		vin = erofs_vmap(compressed_pages, clusterpages);
  192. 	}
  193.  
  194. 	ret = z_erofs_unzip_lz4(vin, vout + pageofs,
  195. 		clusterpages * PAGE_SIZE, llen);
  196. 	if (ret > 0)
  197. 		ret = 0;
  198.  
  199. 	if (!overlapped) {
  200. 		if (clusterpages == 1)
  201. 			kunmap_atomic(vin);
  202. 		else {
  203. 			erofs_vunmap(vin, clusterpages);
  204. 		}
  205. 	} else
  206. 		preempt_enable();
  207.  
  208. 	return ret;
  209. }
  210.

utils.c

  1. // SPDX-License-Identifier: GPL-2.0
  2. /*
  3.  * linux/drivers/staging/erofs/utils.c
  4.  *
  5.  * Copyright (C) 2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13.  
  14. #include "internal.h"
  15. #include <linux/pagevec.h>
  16.  
  17. struct page *erofs_allocpage(struct list_head *pool, gfp_t gfp)
  18. {
  19. 	struct page *page;
  20.  
  21. 	if (!list_empty(pool)) {
  22. 		page = lru_to_page(pool);
  23. 		list_del(&page->lru);
  24. 	} else {
  25. 		page = alloc_pages(gfp | __GFP_NOFAIL, 0);
  26. 	}
  27. 	return page;
  28. }
  29.  
  30. /* global shrink count (for all mounted EROFS instances) */
  31. static atomic_long_t erofs_global_shrink_cnt;
  32.  
  33. #ifdef CONFIG_EROFS_FS_ZIP
  34.  
  35. /* radix_tree and the future XArray both don't use tagptr_t yet */
  36. struct erofs_workgroup *erofs_find_workgroup(
  37. 	struct super_block *sb, pgoff_t index, bool *tag)
  38. {
  39. 	struct erofs_sb_info *sbi = EROFS_SB(sb);
  40. 	struct erofs_workgroup *grp;
  41. 	int oldcount;
  42.  
  43. repeat:
  44. 	rcu_read_lock();
  45. 	grp = radix_tree_lookup(&sbi->workstn_tree, index);
  46. 	if (grp != NULL) {
  47. 		*tag = radix_tree_exceptional_entry(grp);
  48. 		grp = (void *)((unsigned long)grp &
  49. 			~RADIX_TREE_EXCEPTIONAL_ENTRY);
  50.  
  51. 		if (erofs_workgroup_get(grp, &oldcount)) {
  52. 			/* prefer to relax rcu read side */
  53. 			rcu_read_unlock();
  54. 			goto repeat;
  55. 		}
  56.  
  57. 		/* decrease refcount added by erofs_workgroup_put */
  58. 		if (unlikely(oldcount == 1))
  59. 			atomic_long_dec(&erofs_global_shrink_cnt);
  60. 		DBG_BUGON(index != grp->index);
  61. 	}
  62. 	rcu_read_unlock();
  63. 	return grp;
  64. }
  65.  
  66. int erofs_register_workgroup(struct super_block *sb,
  67. 			     struct erofs_workgroup *grp,
  68. 			     bool tag)
  69. {
  70. 	struct erofs_sb_info *sbi;
  71. 	int err;
  72.  
  73. 	/* grp shouldn't be broken or used before */
  74. 	if (unlikely(atomic_read(&grp->refcount) != 1)) {
  75. 		DBG_BUGON(1);
  76. 		return -EINVAL;
  77. 	}
  78.  
  79. 	err = radix_tree_preload(GFP_NOFS);
  80. 	if (err)
  81. 		return err;
  82.  
  83. 	sbi = EROFS_SB(sb);
  84. 	erofs_workstn_lock(sbi);
  85.  
  86. 	if (tag)
  87. 		grp = (void *)((unsigned long)grp |
  88. 			1UL << RADIX_TREE_EXCEPTIONAL_SHIFT);
  89.  
  90. 	/*
  91. 	 * Bump up reference count before making this workgroup
  92. 	 * visible to other users in order to avoid potential UAF
  93. 	 * without serialized by erofs_workstn_lock.
  94. 	 */
  95. 	__erofs_workgroup_get(grp);
  96.  
  97. 	err = radix_tree_insert(&sbi->workstn_tree,
  98. 				grp->index, grp);
  99. 	if (unlikely(err))
  100. 		/*
  101. 		 * it's safe to decrease since the workgroup isn't visible
  102. 		 * and refcount >= 2 (cannot be freezed).
  103. 		 */
  104. 		__erofs_workgroup_put(grp);
  105.  
  106. 	erofs_workstn_unlock(sbi);
  107. 	radix_tree_preload_end();
  108. 	return err;
  109. }
  110.  
  111. extern void erofs_workgroup_free_rcu(struct erofs_workgroup *grp);
  112.  
  113. static void  __erofs_workgroup_free(struct erofs_workgroup *grp)
  114. {
  115. 	atomic_long_dec(&erofs_global_shrink_cnt);
  116. 	erofs_workgroup_free_rcu(grp);
  117. }
  118.  
  119. int erofs_workgroup_put(struct erofs_workgroup *grp)
  120. {
  121. 	int count = atomic_dec_return(&grp->refcount);
  122.  
  123. 	if (count == 1)
  124. 		atomic_long_inc(&erofs_global_shrink_cnt);
  125. 	else if (!count)
  126. 		__erofs_workgroup_free(grp);
  127. 	return count;
  128. }
  129.  
  130. #ifdef EROFS_FS_HAS_MANAGED_CACHE
  131. /* for cache-managed case, customized reclaim paths exist */
  132. static void erofs_workgroup_unfreeze_final(struct erofs_workgroup *grp)
  133. {
  134. 	erofs_workgroup_unfreeze(grp, 0);
  135. 	__erofs_workgroup_free(grp);
  136. }
  137.  
  138. bool erofs_try_to_release_workgroup(struct erofs_sb_info *sbi,
  139. 				    struct erofs_workgroup *grp,
  140. 				    bool cleanup)
  141. {
  142. 	void *entry;
  143.  
  144. 	/*
  145. 	 * for managed cache enabled, the refcount of workgroups
  146. 	 * themselves could be < 0 (freezed). So there is no guarantee
  147. 	 * that all refcount > 0 if managed cache is enabled.
  148. 	 */
  149. 	if (!erofs_workgroup_try_to_freeze(grp, 1))
  150. 		return false;
  151.  
  152. 	/*
  153. 	 * note that all cached pages should be unlinked
  154. 	 * before delete it from the radix tree.
  155. 	 * Otherwise some cached pages of an orphan old workgroup
  156. 	 * could be still linked after the new one is available.
  157. 	 */
  158. 	if (erofs_try_to_free_all_cached_pages(sbi, grp)) {
  159. 		erofs_workgroup_unfreeze(grp, 1);
  160. 		return false;
  161. 	}
  162.  
  163. 	/*
  164. 	 * it is impossible to fail after the workgroup is freezed,
  165. 	 * however in order to avoid some race conditions, add a
  166. 	 * DBG_BUGON to observe this in advance.
  167. 	 */
  168. 	entry = radix_tree_delete(&sbi->workstn_tree, grp->index);
  169. 	DBG_BUGON((void *)((unsigned long)entry &
  170. 			   ~RADIX_TREE_EXCEPTIONAL_ENTRY) != grp);
  171.  
  172. 	/*
  173. 	 * if managed cache is enable, the last refcount
  174. 	 * should indicate the related workstation.
  175. 	 */
  176. 	erofs_workgroup_unfreeze_final(grp);
  177. 	return true;
  178. }
  179.  
  180. #else
  181. /* for nocache case, no customized reclaim path at all */
  182. bool erofs_try_to_release_workgroup(struct erofs_sb_info *sbi,
  183. 				    struct erofs_workgroup *grp,
  184. 				    bool cleanup)
  185. {
  186. 	int cnt = atomic_read(&grp->refcount);
  187. 	void *entry;
  188.  
  189. 	DBG_BUGON(cnt <= 0);
  190. 	DBG_BUGON(cleanup && cnt != 1);
  191.  
  192. 	if (cnt > 1)
  193. 		return false;
  194.  
  195. 	entry = radix_tree_delete(&sbi->workstn_tree, grp->index);
  196. 	DBG_BUGON((void *)((unsigned long)entry &
  197. 			   ~RADIX_TREE_EXCEPTIONAL_ENTRY) != grp);
  198.  
  199. 	/* (rarely) could be grabbed again when freeing */
  200. 	erofs_workgroup_put(grp);
  201. 	return true;
  202. }
  203.  
  204. #endif
  205.  
  206. unsigned long erofs_shrink_workstation(struct erofs_sb_info *sbi,
  207. 				       unsigned long nr_shrink,
  208. 				       bool cleanup)
  209. {
  210. 	pgoff_t first_index = 0;
  211. 	void *batch[PAGEVEC_SIZE];
  212. 	unsigned freed = 0;
  213.  
  214. 	int i, found;
  215. repeat:
  216. 	erofs_workstn_lock(sbi);
  217.  
  218. 	found = radix_tree_gang_lookup(&sbi->workstn_tree,
  219. 		batch, first_index, PAGEVEC_SIZE);
  220.  
  221. 	for (i = 0; i < found; ++i) {
  222. 		struct erofs_workgroup *grp = (void *)
  223. 			((unsigned long)batch[i] &
  224. 				~RADIX_TREE_EXCEPTIONAL_ENTRY);
  225.  
  226. 		first_index = grp->index + 1;
  227.  
  228. 		/* try to shrink each valid workgroup */
  229. 		if (!erofs_try_to_release_workgroup(sbi, grp, cleanup))
  230. 			continue;
  231.  
  232. 		++freed;
  233. 		if (unlikely(!--nr_shrink))
  234. 			break;
  235. 	}
  236. 	erofs_workstn_unlock(sbi);
  237.  
  238. 	if (i && nr_shrink)
  239. 		goto repeat;
  240. 	return freed;
  241. }
  242.  
  243. #endif
  244.  
  245. /* protected by 'erofs_sb_list_lock' */
  246. static unsigned int shrinker_run_no;
  247.  
  248. /* protects the mounted 'erofs_sb_list' */
  249. static DEFINE_SPINLOCK(erofs_sb_list_lock);
  250. static LIST_HEAD(erofs_sb_list);
  251.  
  252. void erofs_register_super(struct super_block *sb)
  253. {
  254. 	struct erofs_sb_info *sbi = EROFS_SB(sb);
  255.  
  256. 	mutex_init(&sbi->umount_mutex);
  257.  
  258. 	spin_lock(&erofs_sb_list_lock);
  259. 	list_add(&sbi->list, &erofs_sb_list);
  260. 	spin_unlock(&erofs_sb_list_lock);
  261. }
  262.  
  263. void erofs_unregister_super(struct super_block *sb)
  264. {
  265. 	spin_lock(&erofs_sb_list_lock);
  266. 	list_del(&EROFS_SB(sb)->list);
  267. 	spin_unlock(&erofs_sb_list_lock);
  268. }
  269.  
  270. unsigned long erofs_shrink_count(struct shrinker *shrink,
  271. 				 struct shrink_control *sc)
  272. {
  273. 	return atomic_long_read(&erofs_global_shrink_cnt);
  274. }
  275.  
  276. unsigned long erofs_shrink_scan(struct shrinker *shrink,
  277. 				struct shrink_control *sc)
  278. {
  279. 	struct erofs_sb_info *sbi;
  280. 	struct list_head *p;
  281.  
  282. 	unsigned long nr = sc->nr_to_scan;
  283. 	unsigned int run_no;
  284. 	unsigned long freed = 0;
  285.  
  286. 	spin_lock(&erofs_sb_list_lock);
  287. 	do
  288. 		run_no = ++shrinker_run_no;
  289. 	while (run_no == 0);
  290.  
  291. 	/* Iterate over all mounted superblocks and try to shrink them */
  292. 	p = erofs_sb_list.next;
  293. 	while (p != &erofs_sb_list) {
  294. 		sbi = list_entry(p, struct erofs_sb_info, list);
  295.  
  296. 		/*
  297. 		 * We move the ones we do to the end of the list, so we stop
  298. 		 * when we see one we have already done.
  299. 		 */
  300. 		if (sbi->shrinker_run_no == run_no)
  301. 			break;
  302.  
  303. 		if (!mutex_trylock(&sbi->umount_mutex)) {
  304. 			p = p->next;
  305. 			continue;
  306. 		}
  307.  
  308. 		spin_unlock(&erofs_sb_list_lock);
  309. 		sbi->shrinker_run_no = run_no;
  310.  
  311. #ifdef CONFIG_EROFS_FS_ZIP
  312. 		freed += erofs_shrink_workstation(sbi, nr - freed, false);
  313. #endif
  314.  
  315. 		spin_lock(&erofs_sb_list_lock);
  316. 		/* Get the next list element before we move this one */
  317. 		p = p->next;
  318.  
  319. 		/*
  320. 		 * Move this one to the end of the list to provide some
  321. 		 * fairness.
  322. 		 */
  323. 		list_move_tail(&sbi->list, &erofs_sb_list);
  324. 		mutex_unlock(&sbi->umount_mutex);
  325.  
  326. 		if (freed >= nr)
  327. 			break;
  328. 	}
  329. 	spin_unlock(&erofs_sb_list_lock);
  330. 	return freed;
  331. }
  332.

xattr.c

  1. // SPDX-License-Identifier: GPL-2.0
  2. /*
  3.  * linux/drivers/staging/erofs/xattr.c
  4.  *
  5.  * Copyright (C) 2017-2018 HUAWEI, Inc.
  6.  *             http://www.huawei.com/
  7.  * Created by Gao Xiang <gaoxiang25@huawei.com>
  8.  *
  9.  * This file is subject to the terms and conditions of the GNU General Public
  10.  * License.  See the file COPYING in the main directory of the Linux
  11.  * distribution for more details.
  12.  */
  13. #include <linux/security.h>
  14. #include "xattr.h"
  15.  
  16. struct xattr_iter {
  17. 	struct super_block *sb;
  18. 	struct page *page;
  19. 	void *kaddr;
  20.  
  21. 	erofs_blk_t blkaddr;
  22. 	unsigned ofs;
  23. };
  24.  
  25. static inline void xattr_iter_end(struct xattr_iter *it, bool atomic)
  26. {
  27. 	/* the only user of kunmap() is 'init_inode_xattrs' */
  28. 	if (unlikely(!atomic))
  29. 		kunmap(it->page);
  30. 	else
  31. 		kunmap_atomic(it->kaddr);
  32.  
  33. 	unlock_page(it->page);
  34. 	put_page(it->page);
  35. }
  36.  
  37. static inline void xattr_iter_end_final(struct xattr_iter *it)
  38. {
  39. 	if (!it->page)
  40. 		return;
  41.  
  42. 	xattr_iter_end(it, true);
  43. }
  44.  
  45. static int init_inode_xattrs(struct inode *inode)
  46. {
  47. 	struct erofs_vnode *const vi = EROFS_V(inode);
  48. 	struct xattr_iter it;
  49. 	unsigned i;
  50. 	struct erofs_xattr_ibody_header *ih;
  51. 	struct erofs_sb_info *sbi;
  52. 	bool atomic_map;
  53. 	int ret = 0;
  54.  
  55. 	/* the most case is that xattrs of this inode are initialized. */
  56. 	if (test_bit(EROFS_V_EA_INITED_BIT, &vi->flags))
  57. 		return 0;
  58.  
  59. 	if (wait_on_bit_lock(&vi->flags, EROFS_V_BL_XATTR_BIT, TASK_KILLABLE))
  60. 		return -ERESTARTSYS;
  61.  
  62. 	/* someone has initialized xattrs for us? */
  63. 	if (test_bit(EROFS_V_EA_INITED_BIT, &vi->flags))
  64. 		goto out_unlock;
  65.  
  66. 	/*
  67. 	 * bypass all xattr operations if ->xattr_isize is not greater than
  68. 	 * sizeof(struct erofs_xattr_ibody_header), in detail:
  69. 	 * 1) it is not enough to contain erofs_xattr_ibody_header then
  70. 	 *    ->xattr_isize should be 0 (it means no xattr);
  71. 	 * 2) it is just to contain erofs_xattr_ibody_header, which is on-disk
  72. 	 *    undefined right now (maybe use later with some new sb feature).
  73. 	 */
  74. 	if (vi->xattr_isize == sizeof(struct erofs_xattr_ibody_header)) {
  75. 		errln("xattr_isize %d of nid %llu is not supported yet",
  76. 		      vi->xattr_isize, vi->nid);
  77. 		ret = -ENOTSUPP;
  78. 		goto out_unlock;
  79. 	} else if (vi->xattr_isize < sizeof(struct erofs_xattr_ibody_header)) {
  80. 		if (unlikely(vi->xattr_isize)) {
  81. 			DBG_BUGON(1);
  82. 			ret = -EIO;
  83. 			goto out_unlock;	/* xattr ondisk layout error */
  84. 		}
  85. 		ret = -ENOATTR;
  86. 		goto out_unlock;
  87. 	}
  88.  
  89. 	sbi = EROFS_I_SB(inode);
  90. 	it.blkaddr = erofs_blknr(iloc(sbi, vi->nid) + vi->inode_isize);
  91. 	it.ofs = erofs_blkoff(iloc(sbi, vi->nid) + vi->inode_isize);
  92.  
  93. 	it.page = erofs_get_inline_page(inode, it.blkaddr);
  94. 	if (IS_ERR(it.page)) {
  95. 		ret = PTR_ERR(it.page);
  96. 		goto out_unlock;
  97. 	}
  98.  
  99. 	/* read in shared xattr array (non-atomic, see kmalloc below) */
  100. 	it.kaddr = kmap(it.page);
  101. 	atomic_map = false;
  102.  
  103. 	ih = (struct erofs_xattr_ibody_header *)(it.kaddr + it.ofs);
  104.  
  105. 	vi->xattr_shared_count = ih->h_shared_count;
  106. 	vi->xattr_shared_xattrs = kmalloc_array(vi->xattr_shared_count,
  107. 						sizeof(uint), GFP_KERNEL);
  108. 	if (!vi->xattr_shared_xattrs) {
  109. 		xattr_iter_end(&it, atomic_map);
  110. 		ret = -ENOMEM;
  111. 		goto out_unlock;
  112. 	}
  113.  
  114. 	/* let's skip ibody header */
  115. 	it.ofs += sizeof(struct erofs_xattr_ibody_header);
  116.  
  117. 	for (i = 0; i < vi->xattr_shared_count; ++i) {
  118. 		if (unlikely(it.ofs >= EROFS_BLKSIZ)) {
  119. 			/* cannot be unaligned */
  120. 			BUG_ON(it.ofs != EROFS_BLKSIZ);
  121. 			xattr_iter_end(&it, atomic_map);
  122.  
  123. 			it.page = erofs_get_meta_page(inode->i_sb,
  124. 				++it.blkaddr, S_ISDIR(inode->i_mode));
  125. 			if (IS_ERR(it.page)) {
  126. 				kfree(vi->xattr_shared_xattrs);
  127. 				vi->xattr_shared_xattrs = NULL;
  128. 				ret = PTR_ERR(it.page);
  129. 				goto out_unlock;
  130. 			}
  131.  
  132. 			it.kaddr = kmap_atomic(it.page);
  133. 			atomic_map = true;
  134. 			it.ofs = 0;
  135. 		}
  136. 		vi->xattr_shared_xattrs[i] =
  137. 			le32_to_cpu(*(__le32 *)(it.kaddr + it.ofs));
  138. 		it.ofs += sizeof(__le32);
  139. 	}
  140. 	xattr_iter_end(&it, atomic_map);
  141.  
  142. 	set_bit(EROFS_V_EA_INITED_BIT, &vi->flags);
  143.  
  144. out_unlock:
  145. 	clear_and_wake_up_bit(EROFS_V_BL_XATTR_BIT, &vi->flags);
  146. 	return ret;
  147. }
  148.  
  149. struct xattr_iter_handlers {
  150. 	int (*entry)(struct xattr_iter *, struct erofs_xattr_entry *);
  151. 	int (*name)(struct xattr_iter *, unsigned, char *, unsigned);
  152. 	int (*alloc_buffer)(struct xattr_iter *, unsigned);
  153. 	void (*value)(struct xattr_iter *, unsigned, char *, unsigned);
  154. };
  155.  
  156. static inline int xattr_iter_fixup(struct xattr_iter *it)
  157. {
  158. 	if (it->ofs < EROFS_BLKSIZ)
  159. 		return 0;
  160.  
  161. 	xattr_iter_end(it, true);
  162.  
  163. 	it->blkaddr += erofs_blknr(it->ofs);
  164. 	it->page = erofs_get_meta_page(it->sb, it->blkaddr, false);
  165. 	if (IS_ERR(it->page)) {
  166. 		int err = PTR_ERR(it->page);
  167.  
  168. 		it->page = NULL;
  169. 		return err;
  170. 	}
  171.  
  172. 	it->kaddr = kmap_atomic(it->page);
  173. 	it->ofs = erofs_blkoff(it->ofs);
  174. 	return 0;
  175. }
  176.  
  177. static int inline_xattr_iter_begin(struct xattr_iter *it,
  178. 	struct inode *inode)
  179. {
  180. 	struct erofs_vnode *const vi = EROFS_V(inode);
  181. 	struct erofs_sb_info *const sbi = EROFS_SB(inode->i_sb);
  182. 	unsigned xattr_header_sz, inline_xattr_ofs;
  183.  
  184. 	xattr_header_sz = inlinexattr_header_size(inode);
  185. 	if (unlikely(xattr_header_sz >= vi->xattr_isize)) {
  186. 		BUG_ON(xattr_header_sz > vi->xattr_isize);
  187. 		return -ENOATTR;
  188. 	}
  189.  
  190. 	inline_xattr_ofs = vi->inode_isize + xattr_header_sz;
  191.  
  192. 	it->blkaddr = erofs_blknr(iloc(sbi, vi->nid) + inline_xattr_ofs);
  193. 	it->ofs = erofs_blkoff(iloc(sbi, vi->nid) + inline_xattr_ofs);
  194.  
  195. 	it->page = erofs_get_inline_page(inode, it->blkaddr);
  196. 	if (IS_ERR(it->page))
  197. 		return PTR_ERR(it->page);
  198.  
  199. 	it->kaddr = kmap_atomic(it->page);
  200. 	return vi->xattr_isize - xattr_header_sz;
  201. }
  202.  
  203. static int xattr_foreach(struct xattr_iter *it,
  204. 	const struct xattr_iter_handlers *op, unsigned int *tlimit)
  205. {
  206. 	struct erofs_xattr_entry entry;
  207. 	unsigned value_sz, processed, slice;
  208. 	int err;
  209.  
  210. 	/* 0. fixup blkaddr, ofs, ipage */
  211. 	err = xattr_iter_fixup(it);
  212. 	if (err)
  213. 		return err;
  214.  
  215. 	/*
  216. 	 * 1. read xattr entry to the memory,
  217. 	 *    since we do EROFS_XATTR_ALIGN
  218. 	 *    therefore entry should be in the page
  219. 	 */
  220. 	entry = *(struct erofs_xattr_entry *)(it->kaddr + it->ofs);
  221. 	if (tlimit != NULL) {
  222. 		unsigned entry_sz = EROFS_XATTR_ENTRY_SIZE(&entry);
  223.  
  224. 		BUG_ON(*tlimit < entry_sz);
  225. 		*tlimit -= entry_sz;
  226. 	}
  227.  
  228. 	it->ofs += sizeof(struct erofs_xattr_entry);
  229. 	value_sz = le16_to_cpu(entry.e_value_size);
  230.  
  231. 	/* handle entry */
  232. 	err = op->entry(it, &entry);
  233. 	if (err) {
  234. 		it->ofs += entry.e_name_len + value_sz;
  235. 		goto out;
  236. 	}
  237.  
  238. 	/* 2. handle xattr name (ofs will finally be at the end of name) */
  239. 	processed = 0;
  240.  
  241. 	while (processed < entry.e_name_len) {
  242. 		if (it->ofs >= EROFS_BLKSIZ) {
  243. 			BUG_ON(it->ofs > EROFS_BLKSIZ);
  244.  
  245. 			err = xattr_iter_fixup(it);
  246. 			if (err)
  247. 				goto out;
  248. 			it->ofs = 0;
  249. 		}
  250.  
  251. 		slice = min_t(unsigned, PAGE_SIZE - it->ofs,
  252. 			entry.e_name_len - processed);
  253.  
  254. 		/* handle name */
  255. 		err = op->name(it, processed, it->kaddr + it->ofs, slice);
  256. 		if (err) {
  257. 			it->ofs += entry.e_name_len - processed + value_sz;
  258. 			goto out;
  259. 		}
  260.  
  261. 		it->ofs += slice;
  262. 		processed += slice;
  263. 	}
  264.  
  265. 	/* 3. handle xattr value */
  266. 	processed = 0;
  267.  
  268. 	if (op->alloc_buffer != NULL) {
  269. 		err = op->alloc_buffer(it, value_sz);
  270. 		if (err) {
  271. 			it->ofs += value_sz;
  272. 			goto out;
  273. 		}
  274. 	}
  275.  
  276. 	while (processed < value_sz) {
  277. 		if (it->ofs >= EROFS_BLKSIZ) {
  278. 			BUG_ON(it->ofs > EROFS_BLKSIZ);
  279.  
  280. 			err = xattr_iter_fixup(it);
  281. 			if (err)
  282. 				goto out;
  283. 			it->ofs = 0;
  284. 		}
  285.  
  286. 		slice = min_t(unsigned, PAGE_SIZE - it->ofs,
  287. 			value_sz - processed);
  288. 		op->value(it, processed, it->kaddr + it->ofs, slice);
  289. 		it->ofs += slice;
  290. 		processed += slice;
  291. 	}
  292.  
  293. out:
  294. 	/* we assume that ofs is aligned with 4 bytes */
  295. 	it->ofs = EROFS_XATTR_ALIGN(it->ofs);
  296. 	return err;
  297. }
  298.  
  299. struct getxattr_iter {
  300. 	struct xattr_iter it;
  301.  
  302. 	char *buffer;
  303. 	int buffer_size, index;
  304. 	struct qstr name;
  305. };
  306.  
  307. static int xattr_entrymatch(struct xattr_iter *_it,
  308. 	struct erofs_xattr_entry *entry)
  309. {
  310. 	struct getxattr_iter *it = container_of(_it, struct getxattr_iter, it);
  311.  
  312. 	return (it->index != entry->e_name_index ||
  313. 		it->name.len != entry->e_name_len) ? -ENOATTR : 0;
  314. }
  315.  
  316. static int xattr_namematch(struct xattr_iter *_it,
  317. 	unsigned processed, char *buf, unsigned len)
  318. {
  319. 	struct getxattr_iter *it = container_of(_it, struct getxattr_iter, it);
  320.  
  321. 	return memcmp(buf, it->name.name + processed, len) ? -ENOATTR : 0;
  322. }
  323.  
  324. static int xattr_checkbuffer(struct xattr_iter *_it,
  325. 	unsigned value_sz)
  326. {
  327. 	struct getxattr_iter *it = container_of(_it, struct getxattr_iter, it);
  328. 	int err = it->buffer_size < value_sz ? -ERANGE : 0;
  329.  
  330. 	it->buffer_size = value_sz;
  331. 	return it->buffer == NULL ? 1 : err;
  332. }
  333.  
  334. static void xattr_copyvalue(struct xattr_iter *_it,
  335. 	unsigned processed, char *buf, unsigned len)
  336. {
  337. 	struct getxattr_iter *it = container_of(_it, struct getxattr_iter, it);
  338.  
  339. 	memcpy(it->buffer + processed, buf, len);
  340. }
  341.  
  342. static const struct xattr_iter_handlers find_xattr_handlers = {
  343. 	.entry = xattr_entrymatch,
  344. 	.name = xattr_namematch,
  345. 	.alloc_buffer = xattr_checkbuffer,
  346. 	.value = xattr_copyvalue
  347. };
  348.  
  349. static int inline_getxattr(struct inode *inode, struct getxattr_iter *it)
  350. {
  351. 	int ret;
  352. 	unsigned remaining;
  353.  
  354. 	ret = inline_xattr_iter_begin(&it->it, inode);
  355. 	if (ret < 0)
  356. 		return ret;
  357.  
  358. 	remaining = ret;
  359. 	while (remaining) {
  360. 		ret = xattr_foreach(&it->it, &find_xattr_handlers, &remaining);
  361. 		if (ret >= 0)
  362. 			break;
  363.  
  364. 		if (ret != -ENOATTR)	/* -ENOMEM, -EIO, etc. */
  365. 			break;
  366. 	}
  367. 	xattr_iter_end_final(&it->it);
  368.  
  369. 	return ret < 0 ? ret : it->buffer_size;
  370. }
  371.  
  372. static int shared_getxattr(struct inode *inode, struct getxattr_iter *it)
  373. {
  374. 	struct erofs_vnode *const vi = EROFS_V(inode);
  375. 	struct erofs_sb_info *const sbi = EROFS_SB(inode->i_sb);
  376. 	unsigned i;
  377. 	int ret = -ENOATTR;
  378.  
  379. 	for (i = 0; i < vi->xattr_shared_count; ++i) {
  380. 		erofs_blk_t blkaddr =
  381. 			xattrblock_addr(sbi, vi->xattr_shared_xattrs[i]);
  382.  
  383. 		it->it.ofs = xattrblock_offset(sbi, vi->xattr_shared_xattrs[i]);
  384.  
  385. 		if (!i || blkaddr != it->it.blkaddr) {
  386. 			if (i)
  387. 				xattr_iter_end(&it->it, true);
  388.  
  389. 			it->it.page = erofs_get_meta_page(inode->i_sb,
  390. 							  blkaddr, false);
  391. 			if (IS_ERR(it->it.page))
  392. 				return PTR_ERR(it->it.page);
  393.  
  394. 			it->it.kaddr = kmap_atomic(it->it.page);
  395. 			it->it.blkaddr = blkaddr;
  396. 		}
  397.  
  398. 		ret = xattr_foreach(&it->it, &find_xattr_handlers, NULL);
  399. 		if (ret >= 0)
  400. 			break;
  401.  
  402. 		if (ret != -ENOATTR)	/* -ENOMEM, -EIO, etc. */
  403. 			break;
  404. 	}
  405. 	if (vi->xattr_shared_count)
  406. 		xattr_iter_end_final(&it->it);
  407.  
  408. 	return ret < 0 ? ret : it->buffer_size;
  409. }
  410.  
  411. static bool erofs_xattr_user_list(struct dentry *dentry)
  412. {
  413. 	return test_opt(EROFS_SB(dentry->d_sb), XATTR_USER);
  414. }
  415.  
  416. static bool erofs_xattr_trusted_list(struct dentry *dentry)
  417. {
  418. 	return capable(CAP_SYS_ADMIN);
  419. }
  420.  
  421. int erofs_getxattr(struct inode *inode, int index,
  422. 	const char *name,
  423. 	void *buffer, size_t buffer_size)
  424. {
  425. 	int ret;
  426. 	struct getxattr_iter it;
  427.  
  428. 	if (unlikely(name == NULL))
  429. 		return -EINVAL;
  430.  
  431. 	ret = init_inode_xattrs(inode);
  432. 	if (ret)
  433. 		return ret;
  434.  
  435. 	it.index = index;
  436.  
  437. 	it.name.len = strlen(name);
  438. 	if (it.name.len > EROFS_NAME_LEN)
  439. 		return -ERANGE;
  440. 	it.name.name = name;
  441.  
  442. 	it.buffer = buffer;
  443. 	it.buffer_size = buffer_size;
  444.  
  445. 	it.it.sb = inode->i_sb;
  446. 	ret = inline_getxattr(inode, &it);
  447. 	if (ret == -ENOATTR)
  448. 		ret = shared_getxattr(inode, &it);
  449. 	return ret;
  450. }
  451.  
  452. static int erofs_xattr_generic_get(const struct xattr_handler *handler,
  453. 		struct dentry *unused, struct inode *inode,
  454. 		const char *name, void *buffer, size_t size)
  455. {
  456. 	struct erofs_sb_info *const sbi = EROFS_I_SB(inode);
  457.  
  458. 	switch (handler->flags) {
  459. 	case EROFS_XATTR_INDEX_USER:
  460. 		if (!test_opt(sbi, XATTR_USER))
  461. 			return -EOPNOTSUPP;
  462. 		break;
  463. 	case EROFS_XATTR_INDEX_TRUSTED:
  464. 		if (!capable(CAP_SYS_ADMIN))
  465. 			return -EPERM;
  466. 		break;
  467. 	case EROFS_XATTR_INDEX_SECURITY:
  468. 		break;
  469. 	default:
  470. 		return -EINVAL;
  471. 	}
  472.  
  473. 	return erofs_getxattr(inode, handler->flags, name, buffer, size);
  474. }
  475.  
  476. const struct xattr_handler erofs_xattr_user_handler = {
  477. 	.prefix	= XATTR_USER_PREFIX,
  478. 	.flags	= EROFS_XATTR_INDEX_USER,
  479. 	.list	= erofs_xattr_user_list,
  480. 	.get	= erofs_xattr_generic_get,
  481. };
  482.  
  483. const struct xattr_handler erofs_xattr_trusted_handler = {
  484. 	.prefix	= XATTR_TRUSTED_PREFIX,
  485. 	.flags	= EROFS_XATTR_INDEX_TRUSTED,
  486. 	.list	= erofs_xattr_trusted_list,
  487. 	.get	= erofs_xattr_generic_get,
  488. };
  489.  
  490. #ifdef CONFIG_EROFS_FS_SECURITY
  491. const struct xattr_handler __maybe_unused erofs_xattr_security_handler = {
  492. 	.prefix	= XATTR_SECURITY_PREFIX,
  493. 	.flags	= EROFS_XATTR_INDEX_SECURITY,
  494. 	.get	= erofs_xattr_generic_get,
  495. };
  496. #endif
  497.  
  498. const struct xattr_handler *erofs_xattr_handlers[] = {
  499. 	&erofs_xattr_user_handler,
  500. #ifdef CONFIG_EROFS_FS_POSIX_ACL
  501. 	&posix_acl_access_xattr_handler,
  502. 	&posix_acl_default_xattr_handler,
  503. #endif
  504. 	&erofs_xattr_trusted_handler,
  505. #ifdef CONFIG_EROFS_FS_SECURITY
  506. 	&erofs_xattr_security_handler,
  507. #endif
  508. 	NULL,
  509. };
  510.  
  511. struct listxattr_iter {
  512. 	struct xattr_iter it;
  513.  
  514. 	struct dentry *dentry;
  515. 	char *buffer;
  516. 	int buffer_size, buffer_ofs;
  517. };
  518.  
  519. static int xattr_entrylist(struct xattr_iter *_it,
  520. 	struct erofs_xattr_entry *entry)
  521. {
  522. 	struct listxattr_iter *it =
  523. 		container_of(_it, struct listxattr_iter, it);
  524. 	unsigned prefix_len;
  525. 	const char *prefix;
  526.  
  527. 	const struct xattr_handler *h =
  528. 		erofs_xattr_handler(entry->e_name_index);
  529.  
  530. 	if (h == NULL || (h->list != NULL && !h->list(it->dentry)))
  531. 		return 1;
  532.  
  533. 	/* Note that at least one of 'prefix' and 'name' should be non-NULL */
  534. 	prefix = h->prefix != NULL ? h->prefix : h->name;
  535. 	prefix_len = strlen(prefix);
  536.  
  537. 	if (it->buffer == NULL) {
  538. 		it->buffer_ofs += prefix_len + entry->e_name_len + 1;
  539. 		return 1;
  540. 	}
  541.  
  542. 	if (it->buffer_ofs + prefix_len
  543. 		+ entry->e_name_len + 1 > it->buffer_size)
  544. 		return -ERANGE;
  545.  
  546. 	memcpy(it->buffer + it->buffer_ofs, prefix, prefix_len);
  547. 	it->buffer_ofs += prefix_len;
  548. 	return 0;
  549. }
  550.  
  551. static int xattr_namelist(struct xattr_iter *_it,
  552. 	unsigned processed, char *buf, unsigned len)
  553. {
  554. 	struct listxattr_iter *it =
  555. 		container_of(_it, struct listxattr_iter, it);
  556.  
  557. 	memcpy(it->buffer + it->buffer_ofs, buf, len);
  558. 	it->buffer_ofs += len;
  559. 	return 0;
  560. }
  561.  
  562. static int xattr_skipvalue(struct xattr_iter *_it,
  563. 	unsigned value_sz)
  564. {
  565. 	struct listxattr_iter *it =
  566. 		container_of(_it, struct listxattr_iter, it);
  567.  
  568. 	it->buffer[it->buffer_ofs++] = '\0';
  569. 	return 1;
  570. }
  571.  
  572. static const struct xattr_iter_handlers list_xattr_handlers = {
  573. 	.entry = xattr_entrylist,
  574. 	.name = xattr_namelist,
  575. 	.alloc_buffer = xattr_skipvalue,
  576. 	.value = NULL
  577. };
  578.  
  579. static int inline_listxattr(struct listxattr_iter *it)
  580. {
  581. 	int ret;
  582. 	unsigned remaining;
  583.  
  584. 	ret = inline_xattr_iter_begin(&it->it, d_inode(it->dentry));
  585. 	if (ret < 0)
  586. 		return ret;
  587.  
  588. 	remaining = ret;
  589. 	while (remaining) {
  590. 		ret = xattr_foreach(&it->it, &list_xattr_handlers, &remaining);
  591. 		if (ret < 0)
  592. 			break;
  593. 	}
  594. 	xattr_iter_end_final(&it->it);
  595. 	return ret < 0 ? ret : it->buffer_ofs;
  596. }
  597.  
  598. static int shared_listxattr(struct listxattr_iter *it)
  599. {
  600. 	struct inode *const inode = d_inode(it->dentry);
  601. 	struct erofs_vnode *const vi = EROFS_V(inode);
  602. 	struct erofs_sb_info *const sbi = EROFS_I_SB(inode);
  603. 	unsigned i;
  604. 	int ret = 0;
  605.  
  606. 	for (i = 0; i < vi->xattr_shared_count; ++i) {
  607. 		erofs_blk_t blkaddr =
  608. 			xattrblock_addr(sbi, vi->xattr_shared_xattrs[i]);
  609.  
  610. 		it->it.ofs = xattrblock_offset(sbi, vi->xattr_shared_xattrs[i]);
  611. 		if (!i || blkaddr != it->it.blkaddr) {
  612. 			if (i)
  613. 				xattr_iter_end(&it->it, true);
  614.  
  615. 			it->it.page = erofs_get_meta_page(inode->i_sb,
  616. 							  blkaddr, false);
  617. 			if (IS_ERR(it->it.page))
  618. 				return PTR_ERR(it->it.page);
  619.  
  620. 			it->it.kaddr = kmap_atomic(it->it.page);
  621. 			it->it.blkaddr = blkaddr;
  622. 		}
  623.  
  624. 		ret = xattr_foreach(&it->it, &list_xattr_handlers, NULL);
  625. 		if (ret < 0)
  626. 			break;
  627. 	}
  628. 	if (vi->xattr_shared_count)
  629. 		xattr_iter_end_final(&it->it);
  630.  
  631. 	return ret < 0 ? ret : it->buffer_ofs;
  632. }
  633.  
  634. ssize_t erofs_listxattr(struct dentry *dentry,
  635. 	char *buffer, size_t buffer_size)
  636. {
  637. 	int ret;
  638. 	struct listxattr_iter it;
  639.  
  640. 	ret = init_inode_xattrs(d_inode(dentry));
  641. 	if (ret == -ENOATTR)
  642. 		return 0;
  643. 	if (ret)
  644. 		return ret;
  645.  
  646. 	it.dentry = dentry;
  647. 	it.buffer = buffer;
  648. 	it.buffer_size = buffer_size;
  649. 	it.buffer_ofs = 0;
  650.  
  651. 	it.it.sb = dentry->d_sb;
  652.  
  653. 	ret = inline_listxattr(&it);
  654. 	if (ret < 0 && ret != -ENOATTR)
  655. 		return ret;
  656. 	return shared_listxattr(&it);
  657. }
  658.

声明：本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：【wpsshop博客】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号