分布式文件系统之MogimenFS的安装使用

Z时代
2024-01-10
分类：技术分享

　　一、简介

　　MogileFS是一个开源的分布式文件存储系统，由LiveJournal旗下的Danga Interactive公司开发；它主要由三部分组成，第一部分是server端，server端主要包括mogilefsd和mogstored两个应用程序。mogilefsd实现的是tracker，它通过数据库来保存元数据信息，包括站点domain、class、host等；mogstored是存储节点(store node)，它其实是个WebDAV服务，默认监听在7500端口，接受客户端的文件存储请求。在MogileFS安装完后，要运行mogadm工具将所有的store node注册到mogilefsd的数据库里，mogilefsd会对这些节点进行管理和监控。第二部是mogilefs的utils,这部分主要是一些工具，比如mogadm、mogupload、mogfileinfo 等等；第三部分是客户端API，mogilefs支持众多编程语言的客户端API接口，使用对应的客户端API接口，我们可以编写mogilefs的客户端，从而实现来管理mogilefs上的文件；

　　二、Mogilefs架构图

　　提示：tracker的主要作用是代理前端用户的请求到数据库查询相关文件的元信息，同时它还要监控注册到tracker的storagenode的健康状态，一旦后端storage挂掉，它会立刻把对应的storage节点上的device从集群信息中删除（对应数据还在，只是在集群的状态信息看不到对应设备），上线时再加入到集群信息；客户端通过向tracker发送请求文件的信息，tracker通过去数据库中找到对应的文件所在storage节点的ip地址、端口、设备id以及文件fid，然后通过http链接的形式从tracker返回给客户端，客户端拿到这tracker的返回的文件地址，然后在去对应的storage请求文件，从而在客户端就访问到对应的文件内容信息；客户端和tracker，storage之间通信都是http协议，从本质上讲storage就是一个web服务器；所以对于客户端我们可以使用web服务器的反向代理，也可以是调用mogilefsAPI写的客户端程序；

　　三、Mogilefs安装部署

　　环境说明

名称	角色	ip地址	端口
node01	tracker+mysql+storage	192.168.0.41	7001/7500/7501/3306
node02	tracker+storage	192.168.0.42	7001/7500/7501
node03	storage	192.168.0.43	7500/7501

　　各server时间同步

　　提示：如果有必要，可以配置各节点用hosts文件解析；

　　安装环境依赖包

[root@node01 ~]# yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO -y
Loaded plugins: fastestmirror
base                                                                                                            | 3.6 kB  00:00:00
epel                                                                                                            | 4.7 kB  00:00:00
extras                                                                                                          | 2.9 kB  00:00:00
updates                                                                                                         | 2.9 kB  00:00:00
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package perl-IO-AIO.x86_64 0:4.20-1.el7 will be installed
--> Processing Dependency: perl(common::sense) for package: perl-IO-AIO-4.20-1.el7.x86_64
---> Package perl-IO-stringy.noarch 0:2.110-22.el7 will be installed
---> Package perl-Net-Netmask.noarch 0:1.9015-13.el7 will be installed
---> Package perl-Sys-Syslog.x86_64 0:0.33-3.el7 will be installed
--> Running transaction check
---> Package perl-common-sense.noarch 0:3.6-4.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=======================================================================================================================================
Package                                Arch                        Version                            Repository                 Size
=======================================================================================================================================
Installing:
perl-IO-AIO                            x86_64                      4.20-1.el7                         epel                      117 k
perl-IO-stringy                        noarch                      2.110-22.el7                       base                       71 k
perl-Net-Netmask                       noarch                      1.9015-13.el7                      epel                       25 k
perl-Sys-Syslog                        x86_64                      0.33-3.el7                         base                       42 k
Installing for dependencies:
perl-common-sense                      noarch                      3.6-4.el7                          epel                       28 k
Transaction Summary
=======================================================================================================================================
Install  4 Packages (+1 Dependent package)
Total download size: 283 k
Installed size: 636 k
Downloading packages:
(1/5): perl-Sys-Syslog-0.33-3.el7.x86_64.rpm                                                                    |  42 kB  00:00:00
(2/5): perl-Net-Netmask-1.9015-13.el7.noarch.rpm                                                                |  25 kB  00:00:00
(3/5): perl-IO-AIO-4.20-1.el7.x86_64.rpm                                                                        | 117 kB  00:00:00
(4/5): perl-common-sense-3.6-4.el7.noarch.rpm                                                                   |  28 kB  00:00:00
(5/5): perl-IO-stringy-2.110-22.el7.noarch.rpm                                                                  |  71 kB  00:00:00
---------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                  642 kB/s | 283 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : perl-common-sense-3.6-4.el7.noarch                                                                                  1/5
Installing : perl-IO-AIO-4.20-1.el7.x86_64                                                                                       2/5
Installing : perl-Net-Netmask-1.9015-13.el7.noarch                                                                               3/5
Installing : perl-Sys-Syslog-0.33-3.el7.x86_64                                                                                   4/5
Installing : perl-IO-stringy-2.110-22.el7.noarch                                                                                 5/5
Verifying  : perl-IO-AIO-4.20-1.el7.x86_64                                                                                       1/5
Verifying  : perl-common-sense-3.6-4.el7.noarch                                                                                  2/5
Verifying  : perl-IO-stringy-2.110-22.el7.noarch                                                                                 3/5
Verifying  : perl-Sys-Syslog-0.33-3.el7.x86_64                                                                                   4/5
Verifying  : perl-Net-Netmask-1.9015-13.el7.noarch                                                                               5/5
Installed:
perl-IO-AIO.x86_64 0:4.20-1.el7           perl-IO-stringy.noarch 0:2.110-22.el7       perl-Net-Netmask.noarch 0:1.9015-13.el7
perl-Sys-Syslog.x86_64 0:0.33-3.el7
Dependency Installed:
perl-common-sense.noarch 0:3.6-4.el7
Complete!
[root@node01 ~]#

　　提示：三个节点都需要安装环境依赖的包；

　　上传rpm包到服务器并安装

[root@node01 ~]# rz

rz waiting to receive.

zmodem trl+C ȡ

100% 1 KB 1 KB/s 00:00:01 0 Errors.el7.centos.noarch.rpm...

100% 173 KB 173 KB/s 00:00:01 0 Errorsentos.noarch.rpm...

100% 26 KB 26 KB/s 00:00:01 0 Errorsentos.noarch.rpm...

100% 75 KB 75 KB/s 00:00:01 0 Errorsh.rpm...

100% 27 KB 27 KB/s 00:00:01 0 Errors.rpm...

100% 29 KB 29 KB/s 00:00:01 0 Errors.noarch.rpm...

100% 262 KB 262 KB/s 00:00:01 0 Errors

[root@node01 ~]# ls

MogileFS-Server-2.46-2.el7.centos.noarch.rpm perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm

MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm

MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm

MogileFS-Utils-2.19-1.el7.centos.noarch.rpm

[root@node01 ~]# yum install ./*.rpm -y

Loaded plugins: fastestmirror

Examining ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-2.46-2.el7.centos.noarch

Marking ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm to be installed

Examining ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch

Marking ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm to be installed

Examining ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogstored-2.46-2.el7.centos.noarch

Marking ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm to be installed

Examining ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm: MogileFS-Utils-2.19-1.el7.centos.noarch

Marking ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm to be installed

Examining ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm: perl-Danga-Socket-1.61-1.el6.rf.noarch

Marking ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm to be installed

Examining ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm: perl-MogileFS-Client-1.14-1.el7.centos.noarch

Marking ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm to be installed

Examining ./perl-Perlbal-1.78-1.el6.noarch.rpm: perl-Perlbal-1.78-1.el6.noarch

Marking ./perl-Perlbal-1.78-1.el6.noarch.rpm to be installed

Resolving Dependencies

--> Running transaction check

---> Package MogileFS-Server.noarch 0:2.46-2.el7.centos will be installed

---> Package MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos will be installed

--> Processing Dependency: perl(DBI) >= 1.44 for package: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch

Loading mirror speeds from cached hostfile

* base: mirrors.aliyun.com

* extras: mirrors.aliyun.com

* updates: mirrors.aliyun.com

--> Processing Dependency: perl(DBD::mysql) >= 3 for package: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch

---> Package MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos will be installed

……省略部分内容……

Installed:

MogileFS-Server.noarch 0:2.46-2.el7.centos MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos

MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos MogileFS-Utils.noarch 0:2.19-1.el7.centos

perl-Danga-Socket.noarch 0:1.61-1.el6.rf perl-MogileFS-Client.noarch 0:1.14-1.el7.centos

perl-Perlbal.noarch 0:1.78-1.el6

Dependency Installed:

mailcap.noarch 0:2.1.41-2.el7 perl-BSD-Resource.x86_64 0:1.29.07-1.el7

perl-Business-ISBN.noarch 0:2.06-2.el7 perl-Business-ISBN-Data.noarch 0:20120719.001-2.el7

perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7 perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7

perl-DBD-MySQL.x86_64 0:4.023-6.el7 perl-DBI.x86_64 0:1.627-4.el7

perl-Data-Dumper.x86_64 0:2.145-3.el7 perl-Digest.noarch 0:1.17-245.el7

perl-Digest-MD5.x86_64 0:2.52-3.el7 perl-Encode-Locale.noarch 0:1.03-5.el7

perl-File-Listing.noarch 0:6.04-7.el7 perl-HTML-Parser.x86_64 0:3.71-4.el7

perl-HTML-Tagset.noarch 0:3.20-15.el7 perl-HTTP-Cookies.noarch 0:6.01-5.el7

perl-HTTP-Daemon.noarch 0:6.01-8.el7 perl-HTTP-Date.noarch 0:6.02-8.el7

perl-HTTP-Message.noarch 0:6.06-6.el7 perl-HTTP-Negotiate.noarch 0:6.01-5.el7

perl-IO-Compress.noarch 0:2.061-2.el7 perl-IO-HTML.noarch 0:1.00-2.el7

perl-IO-Socket-IP.noarch 0:0.21-5.el7 perl-IO-Socket-SSL.noarch 0:1.94-7.el7

perl-LWP-MediaTypes.noarch 0:6.02-2.el7 perl-Mozilla-CA.noarch 0:20130114-5.el7

perl-Net-Daemon.noarch 0:0.48-5.el7 perl-Net-HTTP.noarch 0:6.06-2.el7

perl-Net-LibIDN.x86_64 0:0.12-15.el7 perl-Net-SSLeay.x86_64 0:1.55-6.el7

perl-PlRPC.noarch 0:0.2020-14.el7 perl-Sys-Syscall.noarch 0:0.25-10.el7

perl-TimeDate.noarch 1:2.30-2.el7 perl-URI.noarch 0:1.60-9.el7

perl-WWW-RobotRules.noarch 0:6.02-5.el7 perl-libwww-perl.noarch 0:6.05-2.el7

Complete!

[root@node01 ~]#

　　提示：安装环境依赖和以上rpm包，首先要配置好epel源；因为有些依赖包，它来自epel源；相关包我已经上传至我的网盘http://www.qiuhom.com/d/383f34f57cf847beb296/；

　　安装mariadb-server

　　配置yum源

[root@node01 ~]# cat /etc/yum.repos.d/mairadb.repo
[mariadb]
name=mariadb repo
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mariadb//mariadb-10.1.46/yum/centos/7/x86_64/
gpgcheck=0
[root@node01 ~]#

　　安装mariadb-server

[root@node01 ~]# yum install MariaDB-server -y

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

* base: mirrors.aliyun.com

* extras: mirrors.aliyun.com

* updates: mirrors.aliyun.com

Resolving Dependencies

--> Running transaction check

---> Package MariaDB-server.x86_64 0:10.1.46-1.el7.centos will be installed

--> Processing Dependency: MariaDB-client for package: MariaDB-server-10.1.46-1.el7.centos.x86_64

--> Processing Dependency: rsync for package: MariaDB-server-10.1.46-1.el7.centos.x86_64

--> Processing Dependency: MariaDB-common for package: MariaDB-server-10.1.46-1.el7.centos.x86_64

--> Processing Dependency: galera for package: MariaDB-server-10.1.46-1.el7.centos.x86_64

--> Processing Dependency: libjemalloc.so.1()(64bit) for package: MariaDB-server-10.1.46-1.el7.centos.x86_64

--> Running transaction check

……省略部分内容……

Installed:

MariaDB-server.x86_64 0:10.1.46-1.el7.centos MariaDB-shared.x86_64 0:10.1.46-1.el7.centos

Dependency Installed:

MariaDB-client.x86_64 0:10.1.46-1.el7.centos MariaDB-common.x86_64 0:10.1.46-1.el7.centos

boost-program-options.x86_64 0:1.53.0-28.el7 galera.x86_64 0:25.3.29-1.rhel7.el7.centos

jemalloc.x86_64 0:3.6.0-1.el7 rsync.x86_64 0:3.1.2-10.el7

Replaced:

mariadb-libs.x86_64 1:5.5.56-2.el7

Complete!

[root@node01 ~]#

　　提示：mariadb-server包名是大写的MariaDB-server；

　　配置mariadb忽略名称解析

　　设置mariadb开机自动启动，并启动mariadb

　　添加用于mogilefs访问数据库的账号

　　初始化数据库

[root@node01 ~]# mogdbsetup --help
Usage: mogdbsetup [opts]
Options:
Default      Description
============ ===========================================
--verbose        <off>        Be verbose about what's happening.
--dbhost=        localhost    hostname or IP to database server.
--dbport=        dbd default  port number to database server.
--dbname=        mogilefs     database name to create/upgrade.
--dbrootuser=    root         Database administrator username.  Only needed
for initial setup, not subsequent upgrades.
--dbrootpass=    <blank>      Database administrator password.  Only needed
for initial setup, not subsequent upgrades.
--dbuser=        mogile       Regular database user to create and/or use
for MogileFS database.  This is what the
mogilefsd trackers connect as.
--dbpass=        <blank>      You should change this, especially if your
database servers are accessible to other users
on the network.  But they shouldn't be
if you're running MogileFS, because MogileFS
assumes your network is closed.
--type=         MySQL        Which MogileFS::Store implementation to use.
Available: MySQL, Postgres
--yes                        Run without questions.
[root@node01 ~]# mogdbsetup --dbhost=192.168.0.41 --dbport=3306 --dbname=mogilefs --dbrootpass=admin --dbuser=mogilefs --dbpass=mogilefs
This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information.  Run with --yes to shut up these prompts.
Continue? [N/y]: y
Create/Upgrade database name 'mogilefs'? [Y/n]: y
Grant all privileges to user 'mogilefs', connecting from anywhere, to the mogilefs database 'mogilefs'? [Y/n]: n
Stopped.
[root@node01 ~]# mogdbsetup --dbhost=192.168.0.41 --dbport=3306 --dbname=mogilefs --dbrootpass=admin --dbuser=mogilefs --dbpass=mogilefs
This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information.  Run with --yes to shut up these prompts.
Continue? [N/y]: y
[root@node01 ~]#

　　提示：如果不想让mogdbsetup在数据库中创建一个任何地址都可访问的mogilefs的用户，在最后的选项中使用n即可，然后在重复执行上述命令即可；如果不想交互式输入y，也可以直接在命令后面用--yes；

　　验证：在数据库中查看对应的库是否都已创建，对应库中是否都有表？

[root@node01 ~]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7
Server version: 10.1.46-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mogilefs           |
| mysql              |
| performance_schema |
| test               |
+--------------------+
5 rows in set (0.01 sec)
MariaDB [(none)]> use mogilefs
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [mogilefs]> show tables;
+----------------------+
| Tables_in_mogilefs   |
+----------------------+
| checksum             |
| class                |
| device               |
| domain               |
| file                 |
| file_on              |
| file_on_corrupt      |
| file_to_delete       |
| file_to_delete2      |
| file_to_delete_later |
| file_to_queue        |
| file_to_replicate    |
| fsck_log             |
| host                 |
| server_settings      |
| tempfile             |
| unreachable_fids     |
+----------------------+
17 rows in set (0.00 sec)
MariaDB [mogilefs]>

　　到此数据库就准备好了

　　配置mogilefsd连接数据库

　　提示：我们只需要修改红框中的内容即可；db_dsn用于指定连接数据库的驱动，以及数据库库名和主机地址；db_user用于指定连接数据库所使用的用户，db_pass用于指定连接数据库的密码；listen用于指定mogilefsd监听的ip地址和端口；后面的配置就是和启动的进程数量的配置，这个可以根据业务的规模来设置；

　　启动mogilefsd

　　提示：在centos7上启动mogilefsd时，会报错，我们可以不用管这个错误，实际上mogilefsd是启动成功的，只要看到对应的端口和进程起来，表示mogilefsd启动就没有问题；

　　配置node01上的mogstored

[root@node01 ~]# cat /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /data/mogilefs
[root@node01 ~]# mkdir /data/mogilefs/ -pv
mkdir: created directory ‘/data’
mkdir: created directory ‘/data/mogilefs/’
[root@node01 ~]# chown -R mogilefs.mogilefs /data/mogilefs/
[root@node01 ~]# ll -d /data/mogilefs/
drwxr-xr-x 2 mogilefs mogilefs 6 Sep 16 13:37 /data/mogilefs/
[root@node01 ~]#

　　启动mogstored

　　提示：在centos7上启动mogstored也会同启动mogilefsd一样报错，这个主要是因为systemd的缘故，在centos6不会报错；到此，node01上的mariadb、mogilefsd和mogstored都已经安装配置好了；

　　复制安装包到node02和node03

　　在node02上安装

[root@node02 ~]# yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO ./*.rpm -y

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

* base: mirrors.aliyun.com

* extras: mirrors.aliyun.com

* updates: mirrors.aliyun.com

Package perl-Net-Netmask-1.9015-13.el7.noarch already installed and latest version

Package perl-IO-stringy-2.110-22.el7.noarch already installed and latest version

Package perl-Sys-Syslog-0.33-3.el7.x86_64 already installed and latest version

Package perl-IO-AIO-4.20-1.el7.x86_64 already installed and latest version