20141113

아침 8시 50분에 tonto, sphinx3, calculix, dealII이 여전히 실행 중이었다. NAS parallel benchmark 중에서는 bt.C.x가 여전히 실행 중이었다. 우선 지금까지 구해진 BBV로 fast forwarding할 instruction count를 계산했다. 그리고 몇몇 워크로드의 fast forwarding할 instruction count를 약간 수정했다.

SPECINT
perlbench_checkspam : 357 simulation point
perlbench_diffmail : 955 simulation point
perlbench_splitmail : 571 simulation point
bzip2_source : 7 simulation point
bzip2_program : 189 simulation point
bzip2_combined : 182 simulation point
bzip2_chicken : 70 simulation point
bzip2_liberty : 498 simulation point
bzip2_html : 136 simulation point
gcc_166 : 126 simulation point
gcc_200 : 112 simulation point
gcc_typecheck : 161 simulation point
gcc_decl : 17 simulation point
gcc_expr : 2 simulation point
gcc_expr2 : 4 simulation point
gcc_g23 : 2 simulation point
gcc_s04 : 37 simulation point
gcc_scilab : 26 simulation point
mcf : 133 simulation point
gobmk_1313 : 130 simulation point
gobmk_nngs : 221 simulation point
gobmk_score : 217 simulation point
gobmk_trevorc : 995 simulation point
gobmk_trevord : 146 simulation point
hmmer_nph3 : 2842 simulation point
hmmer_retro : 2296 simulation point
sjeng : 1967 simulation point
libquantum : 1623 simulation point
h264ref_foreman_base : 501 simulation point
h264ref_foreman_main : 133 simulation point
h264ref_sss_main : 808 simulation point
omnetpp : 1894 simulation point
astar_biglake : 177 simulation point
astar_river : 39 simulation point
xalancbmk : 605 simulation point
 
SPECFP
bwaves : 312 simulation point
gamess_cytosine : 57 simulation point
gamess_gradient : 22 simulation point
gamess_triazolium : 90 simulation point
milc : 112 simulation point
zeusmp : 100 simulation point #valgrind error, selected arbitrary proper instruction count
gromacs : 1984 simulation point
cactusADM : 1080 simulation point
leslie3d : 796 simulation point
namd : 33 simulation point
dealII : 123 simulation point #not completed yet
soplex_pds : 331 simulation point
soplex_ref : 272 simulation point
povray : 370 simulation point
calculix : 398 simulation point #not completed yet
GemsFDTD : 244 simulation point
tonto : 336 simulation point #not completed yet
lbm : 1088 simulation point
wrf : 1166 simulation point
sphinx3 : 1948 simulation point #not completed yet

SPECCPU2006에 대해 checkpoint를 생성하는 중.

checkpoint_after_SPEC.py

#! /usr/bin/python
import os
import subprocess
import sys

# Set up default variables -------------------------------------------------------#
login_cmds = ["ID\n", "PASSWD\n"]     #FIXME
cwd = os.getcwd()
qemu_bin = '%s/qemu/qemu-system-x86_64' % cwd
qemu_img = '/path/to/marss/img.qcow2' #FIXME
vm_memory = 4096
qemu_cmd = ''
#---------------------------------------------------------------------------------#

def add_to_cmd(opt):
    global qemu_cmd
    qemu_cmd = "%s %s" % (qemu_cmd, opt)

def pty_to_stdout(fd, untill_chr):
    chr = '1'
    while chr != untill_chr:
        try:
		    chr = os.read(fd, 1)
        except OSError:
            chr = untill_chr
            print "OSError"	
        else:
     	    sys.stdout.write(chr)
    sys.stdout.flush()

def pty_login(fd):
    os.write(fd, login_cmds[0])
    pty_to_stdout(fd, ':')
    os.write(fd, login_cmds[1])

# Generate a common command string
add_to_cmd(qemu_bin)
add_to_cmd('-m %d' % vm_memory)
add_to_cmd('-serial pty')
add_to_cmd('-vnc :22')
add_to_cmd('-cpu core2duo')
add_to_cmd('-drive file=%s,cache=unsafe' % qemu_img)

# Checkpoint list
check_list = []

# SPEC 2006 Benchmarks
spec_list = [
            #SPECINT2006
            {'name' : 'perlbench_checkspam',
             'command' :
             '''checkpoint_after 35750608000 perlbench_checkspam; python ~/runspec.py perlbench_checkspam; kill_sim
             '''},
            {'name' : 'perlbench_diffmail',
             'command' :
             '''checkpoint_after 95550608000 perlbench_diffmail; python ~/runspec.py perlbench_diffmail; kill_sim
             '''},
            {'name' : 'perlbench_splitmail',
             'command' :
             '''checkpoint_after 57150608000 perlbench_splitmail; python ~/runspec.py perlbench_splitmail; kill_sim
             '''},
            {'name' : 'bzip2_source',
             'command' :
             '''checkpoint_after 750608000 bzip2_source; python ~/runspec.py bzip2_source; kill_sim
             '''},
            {'name' : 'bzip2_program',
             'command' :
             '''checkpoint_after 18950608000 bzip2_program; python ~/runspec.py bzip2_program; kill_sim
             '''},
            {'name' : 'bzip2_combined',
             'command' :
             '''checkpoint_after 18250608000 bzip2_combined; python ~/runspec.py bzip2_combined; kill_sim
             '''},
            {'name' : 'bzip2_chicken',
             'command' :
             '''checkpoint_after 7050608000 bzip2_chicken; python ~/runspec.py bzip2_chicken; kill_sim
             '''},
            {'name' : 'bzip2_liberty',
             'command' :
             '''checkpoint_after 49850608000 bzip2_liberty; python ~/runspec.py bzip2_liberty; kill_sim
             '''},
            {'name' : 'bzip2_html',
             'command' :
             '''checkpoint_after 13650608000 bzip2_html; python ~/runspec.py bzip2_html; kill_sim
             '''},
            {'name' : 'gcc_166',
             'command' :
             '''checkpoint_after 12650608000 gcc_166; python ~/runspec.py gcc_166; kill_sim
             '''},
            {'name' : 'gcc_200',
             'command' :
             '''checkpoint_after 11250608000 gcc_200; python ~/runspec.py gcc_200; kill_sim
             '''},
            {'name' : 'gcc_typecheck',
             'command' :
             '''checkpoint_after 16150608000 gcc_typecheck; python ~/runspec.py gcc_typecheck; kill_sim
             '''},
            {'name' : 'gcc_decl',
             'command' :
             '''checkpoint_after 1750608000 gcc_decl; python ~/runspec.py gcc_decl; kill_sim
             '''},
            {'name' : 'gcc_expr',
             'command' :
             '''checkpoint_after 250608000 gcc_expr; python ~/runspec.py gcc_expr; kill_sim
             '''},
            {'name' : 'gcc_expr2',
             'command' :
             '''checkpoint_after 450608000 gcc_expr2; python ~/runspec.py gcc_expr2; kill_sim
             '''},
            {'name' : 'gcc_g23',
             'command' :
             '''checkpoint_after 250608000 gcc_g23; python ~/runspec.py gcc_g23; kill_sim
             '''},
            {'name' : 'gcc_s04',
             'command' :
             '''checkpoint_after 3750608000 gcc_s04; python ~/runspec.py gcc_s04; kill_sim
             '''},
            {'name' : 'gcc_scilab',
             'command' :
             '''checkpoint_after 2650608000 gcc_scilab; python ~/runspec.py gcc_scilab; kill_sim
             '''},
            {'name' : 'mcf',
             'command' :
             '''checkpoint_after 13350608000 mcf; python ~/runspec.py mcf; kill_sim
             '''},
            {'name' : 'gobmk_1313',
             'command' :
             '''checkpoint_after 13050608000 gobmk_1313; python ~/runspec.py gobmk_1313; kill_sim
             '''},
            {'name' : 'gobmk_nngs',
             'command' :
             '''checkpoint_after 22150608000 gobmk_nngs; python ~/runspec.py gobmk_nngs; kill_sim
             '''},
            {'name' : 'gobmk_score',
             'command' :
             '''checkpoint_after 21750608000 gobmk_score; python ~/runspec.py gobmk_score; kill_sim
             '''},
            {'name' : 'gobmk_trevorc',
             'command' :
             '''checkpoint_after 99550608000 gobmk_trevorc; python ~/runspec.py gobmk_trevorc; kill_sim
             '''},
            {'name' : 'gobmk_trevord',
             'command' :
             '''checkpoint_after 14650608000 gobmk_trevord; python ~/runspec.py gobmk_trevord; kill_sim
             '''},
            {'name' : 'hmmer_nph3',
             'command' :
             '''checkpoint_after 284250608000 hmmer_nph3; python ~/runspec.py hmmer_nph3; kill_sim
             '''},
            {'name' : 'hmmer_retro',
             'command' :
             '''checkpoint_after 229650608000 hmmer_retro; python ~/runspec.py hmmer_retro; kill_sim
             '''},
            {'name' : 'sjeng',
             'command' :
             '''checkpoint_after 196750608000 sjeng; python ~/runspec.py sjeng; kill_sim
             '''},
            {'name' : 'libquantum',
             'command' :
             '''checkpoint_after 162350608000 libquantum; python ~/runspec.py libquantum; kill_sim
             '''},
            {'name' : 'h264ref_foreman_base',
             'command' :
             '''checkpoint_after 50150608000 h264ref_foreman_base; python ~/runspec.py h264ref_foreman_base; kill_sim
             '''},
            {'name' : 'h264ref_foreman_main',
             'command' :
             '''checkpoint_after 13350608000 h264ref_foreman_main; python ~/runspec.py h264ref_foreman_main; kill_sim
             '''},
            {'name' : 'h264ref_sss_main',
             'command' :
             '''checkpoint_after 80850608000 h264ref_sss_main; python ~/runspec.py h264ref_sss_main; kill_sim
             '''},
            {'name' : 'omnetpp',
             'command' :
             '''checkpoint_after 189450608000 omnetpp; python ~/runspec.py omnetpp; kill_sim
             '''},
            {'name' : 'astar_biglake',
             'command' :
             '''checkpoint_after 17750608000 astar_biglake; python ~/runspec.py astar_biglake; kill_sim
             '''},
            {'name' : 'astar_river',
             'command' :
             '''checkpoint_after 3950608000 astar_river; python ~/runspec.py astar_river; kill_sim
             '''},
            {'name' : 'xalancbmk',
             'command' :
             '''checkpoint_after 60550608000 xalancbmk; python ~/runspec.py xalancbmk; kill_sim
             '''},
            #SPECFP2006
            {'name' : 'bwaves',
             'command' :
             '''checkpoint_after 31250608000 bwaves; python ~/runspec.py bwaves; kill_sim
             '''},
            {'name' : 'gamess_cytosine',
             'command' :
             '''checkpoint_after 5750608000 gamess_cytosine; python ~/runspec.py gamess_cytosine; kill_sim
             '''},
            {'name' : 'gamess_gradient',
             'command' :
             '''checkpoint_after 2250608000 gamess_gradient; python ~/runspec.py gamess_gradient; kill_sim
             '''},
            {'name' : 'gamess_triazolium',
             'command' :
             '''checkpoint_after 9050608000 gamess_triazolium; python ~/runspec.py gamess_triazolium; kill_sim
             '''},
            {'name' : 'milc',
             'command' :
             '''checkpoint_after 11250608000 milc; python ~/runspec.py milc; kill_sim
             '''},
            {'name' : 'zeusmp',
             'command' :
             '''checkpoint_after 10050608000 zeusmp; python ~/runspec.py zeusmp; kill_sim
             '''},
            {'name' : 'gromacs',
             'command' :
             '''checkpoint_after 198450608000 gromacs; python ~/runspec.py gromacs; kill_sim
             '''},
            {'name' : 'cactusADM',
             'command' :
             '''checkpoint_after 108050608000 cactusADM; python ~/runspec.py cactusADM; kill_sim
             '''},
            {'name' : 'leslie3d',
             'command' :
             '''checkpoint_after 79650608000 leslie3d; python ~/runspec.py leslie3d; kill_sim
             '''},
            {'name' : 'namd',
             'command' :
             '''checkpoint_after 3350608000 namd; python ~/runspec.py namd; kill_sim
             '''},
            {'name' : 'dealII',
             'command' :
             '''checkpoint_after 12350608000 dealII; python ~/runspec.py dealII; kill_sim
             '''},
            {'name' : 'soplex_pds',
             'command' :
             '''checkpoint_after 33150608000 soplex_pds; python ~/runspec.py soplex_pds; kill_sim
             '''},
            {'name' : 'soplex_ref',
             'command' :
             '''checkpoint_after 27250608000 soplex_ref; python ~/runspec.py soplex_ref; kill_sim
             '''},
            {'name' : 'povray',
             'command' :
             '''checkpoint_after 37050608000 povray; python ~/runspec.py povray; kill_sim
             '''},
            {'name' : 'calculix',
             'command' :
             '''checkpoint_after 39850608000 calculix; python ~/runspec.py calculix; kill_sim
             '''},
            {'name' : 'GemsFDTD',
             'command' :
             '''checkpoint_after 24450608000 GemsFDTD; python ~/runspec.py GemsFDTD; kill_sim
             '''},
            {'name' : 'tonto',
             'command' :
             '''checkpoint_after 33650608000 tonto; python ~/runspec.py tonto; kill_sim
             '''},
            {'name' : 'lbm',
             'command' :
             '''checkpoint_after 108850608000 lbm; python ~/runspec.py lbm; kill_sim
             '''},
            {'name' : 'wrf',
             'command' :
             '''checkpoint_after 116650608000 wrf; python ~/runspec.py wrf; kill_sim
             '''},
            {'name' : 'sphinx3',
             'command' :
             '''checkpoint_after 194850608000 sphinx3; python ~/runspec.py sphinx3; kill_sim
             '''}
        ]

# To create all spec checkpoints
check_list = spec_list

print("Execution command: %s" % qemu_cmd)
print("Number of Chekcpoints to create: %d" % len(check_list))

# Start simulation from checkpoints
pty_prefix = 'char device redirected to '
for checkpoint in check_list:

    print("Starting QEMU for checkpoint: %s" % checkpoint['name'])

    p = subprocess.Popen(qemu_cmd.split(), stdout=subprocess.PIPE,\
						 stderr=subprocess.STDOUT, bufsize=0)

    pty_term = None

    while p.poll() is None:
        line = p.stdout.readline()
        sys.stdout.write(line)
        if line.startswith(pty_prefix):
            dev_name = line[len(pty_prefix):].strip()
            # Open the device terminal and send simulation command
            pty_term = os.open(dev_name, os.O_RDWR)
            break

    if pty_term == None:
        print("ERROR: While connecting with pty terminal")
        continue

    pty_to_stdout(pty_term, ':')
    # Now send the login commands to the termianl and wait
    # untill some response text
    pty_login(pty_term)
    pty_to_stdout(pty_term, '#')
    # At this point we assume that we have successfully logged in
    # Now give the command to create checkpoint
    os.write(pty_term, checkpoint['command'])
    pty_to_stdout(pty_term, '#')
#    sys.stdout.write('\n')
#    for line in p.stdout:
#        sys.stdout.write(line)

    # Wait for simulation to complete
    p.wait()

10시 30분 perlbench_splitmail 실행. 12시 gcc_expr 실행. 1시 gcc_expr2 실행. 2시 20분에도 여전히 gcc_expr2 실행 중이기에 확인해보니 부팅 중 멈춤. 재시작 함. 3시에 gobmk_score 실행. 4시 15분에 hmmer_retro 실행.

graph500, gups, NAS parallel benchmark의 simulation point는 다음과 같다.

graph500 : 617 simulation point
gups : 1 simulation point
bt.C.x : 10128 #not completed yet
cg.C.x : 123 simulation point
dc.B.x : 85 simulation point
ep.C.x : 2752 simulation point
ft.C.x : 2000 simulation point #valgrind error, selected arbitrary proper instruction count
is.C.x : 2000 simulation point #valgrind error, selected arbitrary proper instruction count
lu.C.x : 2130 simulation point
mg.C.x : 2000 simulation point #valgrind error, selected arbitrary proper instruction count
sp.C.x : 2480 simulation point 
ua.C.x : 7189 simulation point 

graph500, gups, NPB에 대해 checkpoint를 생성하는 중.

checkpoint_after_NPB.py

#! /usr/bin/python
import os
import subprocess
import sys

# Set up default variables -------------------------------------------------------#
login_cmds = ["ID\n", "PASSWD\n"]		 #FIXME
cwd = os.getcwd()
qemu_bin = '%s/qemu/qemu-system-x86_64' % cwd
qemu_img = '/path/to/marss/img.qcow2'    #FIXME
vm_memory = 4096
qemu_cmd = ''
#---------------------------------------------------------------------------------#

def add_to_cmd(opt):
    global qemu_cmd
    qemu_cmd = "%s %s" % (qemu_cmd, opt)

def pty_to_stdout(fd, untill_chr):
    chr = '1'
    while chr != untill_chr:
        try:
	    chr = os.read(fd, 1)
        except OSError:
            chr = untill_chr
            print "OSError"	
        else:
     	    sys.stdout.write(chr)
    sys.stdout.flush()

def pty_login(fd):
    os.write(fd, login_cmds[0])
    pty_to_stdout(fd, ':')
    os.write(fd, login_cmds[1])

# Generate a common command string
add_to_cmd(qemu_bin)
add_to_cmd('-m %d' % vm_memory)
add_to_cmd('-serial pty')
add_to_cmd('-vnc :33')
add_to_cmd('-cpu core2duo')
add_to_cmd('-drive file=%s,cache=unsafe' % qemu_img)

# Checkpoint list
check_list = []

benchmark_list = [
            {'name' : 'graph500',
             'command' :
             '''checkpoint_after 61700000000 graph500; ~/graph500/seq-csr/seq-csr -s 22 -e 16 > /dev/null ; kill_sim
             '''},
            {'name' : 'gups',
             'command' :
             '''checkpoint_after 100000000 gups; ~/gups/gups_vanilla 28 100000 4096 > /dev/null; kill_sim
             '''},
            {'name' : 'NPB_BT',
             'command' :
             '''checkpoint_after 1012800000000 NPB_BT; ~/NPB3.3.1/NPB3.3-SER/bin/bt.C.x > /dev/null; kill_sim
             '''},
            {'name' : 'NPB_CG',
             'command' :
             '''checkpoint_after 12300000000 NPB_CG; ~/NPB3.3.1/NPB3.3-SER/bin/cg.C.x > /dev/null; kill_sim
             '''},
            {'name' : 'NPB_DC',
             'command' :
             '''checkpoint_after 8500000000 NPB_DC; ~/NPB3.3.1/NPB3.3-SER/bin/dc.B.x > /dev/null; kill_sim
             '''},
            {'name' : 'NPB_EP',
             'command' :
             '''checkpoint_after 275200000000 NPB_EP; ~/NPB3.3.1/NPB3.3-SER/bin/ep.C.x > /dev/null; kill_sim
             '''},
            {'name' : 'NPB_FT',
             'command' :
             '''checkpoint_after 200000000000 NPB_FT; ~/NPB3.3.1/NPB3.3-SER/bin/ft.C.x > /dev/null; kill_sim
             '''},
            {'name' : 'NPB_IS',
             'command' :
             '''checkpoint_after 200000000000 NPB_IS; ~/NPB3.3.1/NPB3.3-SER/bin/is.C.x > /dev/null; kill_sim
             '''},
            {'name' : 'NPB_LU',
             'command' :
             '''checkpoint_after 213000000000 NPB_LU; ~/NPB3.3.1/NPB3.3-SER/bin/lu.C.x > /dev/null; kill_sim
             '''},
            {'name' : 'NPB_MG',
             'command' :
             '''checkpoint_after 200000000000 NPB_MG; ~/NPB3.3.1/NPB3.3-SER/bin/mg.C.x > /dev/null; kill_sim
             '''},
            {'name' : 'NPB_SP',
             'command' :
             '''checkpoint_after 248000000000 NPB_SP; ~/NPB3.3.1/NPB3.3-SER/bin/sp.C.x > /dev/null; kill_sim
             '''},
            {'name' : 'NPB_UA',
             'command' :
             '''checkpoint_after 718900000000 NPB_UA; ~/NPB3.3.1/NPB3.3-SER/bin/ua.C.x > /dev/null; kill_sim
             '''}
        ]

# To create all spec checkpoints
check_list = benchmark_list

print("Execution command: %s" % qemu_cmd)
print("Number of Chekcpoints to create: %d" % len(check_list))

# Start simulation from checkpoints
pty_prefix = 'char device redirected to '
for checkpoint in check_list:

    print("Starting QEMU for checkpoint: %s" % checkpoint['name'])

    p = subprocess.Popen(qemu_cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=0)

    pty_term = None

    while p.poll() is None:
        line = p.stdout.readline()
        sys.stdout.write(line)
        if line.startswith(pty_prefix):
            dev_name = line[len(pty_prefix):].strip()
            # Open the device terminal and send simulation command
            pty_term = os.open(dev_name, os.O_RDWR)
            break

    if pty_term == None:
        print("ERROR: While connecting with pty terminal")
        continue

    pty_to_stdout(pty_term, ':')
    # Now send the login commands to the termianl and wait
    # untill some response text
    pty_login(pty_term)
    pty_to_stdout(pty_term, '#')
    # At this point we assume that we have successfully logged in
    # Now give the command to create checkpoint
    os.write(pty_term, checkpoint['command'])
    pty_to_stdout(pty_term, '#')
#    sys.stdout.write('\n')
#    for line in p.stdout:
#        sys.stdout.write(line)

    # Wait for simulation to complete
    p.wait()

12시 graph500 실행. 12시 40분 NPB BT 실행. 2시 20분 NPB IS 실행. 3시에도 여전히 NPB IS 실행 중이기에 확인해보니 부팅 중 멈춤. 재시작 함. 오후 4시, 스크립트가 잘못 짜여 있었다. 출력을 /dev/null로 보내는 것을 빠뜨림. 스크립트를 수정하고, 처음부터 다시 시작함.

4시 15분에 확인해보니, 아침에 실행중이던 tonto, sphinx3, calculix, dealII의 BBV 생성이 끝나있었다. SimPointing해보았다. 다른 것은 그대로 써도 되고, sphinx3의 simulation point는 322로 바뀌어야 한다.

SPECINT
perlbench_checkspam : 357 simulation point
perlbench_diffmail : 955 simulation point
perlbench_splitmail : 571 simulation point
bzip2_source : 7 simulation point
bzip2_program : 189 simulation point
bzip2_combined : 182 simulation point
bzip2_chicken : 70 simulation point
bzip2_liberty : 498 simulation point
bzip2_html : 136 simulation point
gcc_166 : 126 simulation point
gcc_200 : 112 simulation point
gcc_typecheck : 161 simulation point
gcc_decl : 17 simulation point
gcc_expr : 2 simulation point
gcc_expr2 : 4 simulation point
gcc_g23 : 2 simulation point
gcc_s04 : 37 simulation point
gcc_scilab : 26 simulation point
mcf : 133 simulation point
gobmk_1313 : 130 simulation point
gobmk_nngs : 221 simulation point
gobmk_score : 217 simulation point
gobmk_trevorc : 995 simulation point
gobmk_trevord : 146 simulation point
hmmer_nph3 : 2842 simulation point
hmmer_retro : 2296 simulation point
sjeng : 1967 simulation point
libquantum : 1623 simulation point
h264ref_foreman_base : 501 simulation point
h264ref_foreman_main : 133 simulation point
h264ref_sss_main : 808 simulation point
omnetpp : 1894 simulation point
astar_biglake : 177 simulation point
astar_river : 39 simulation point
xalancbmk : 605 simulation point
 
SPECFP
bwaves : 312 simulation point
gamess_cytosine : 57 simulation point
gamess_gradient : 22 simulation point
gamess_triazolium : 90 simulation point
milc : 112 simulation point
zeusmp : 100 simulation point #valgrind error, selected arbitrary proper instruction count
gromacs : 1984 simulation point
cactusADM : 1080 simulation point
leslie3d : 796 simulation point
namd : 33 simulation point
dealII : 123 simulation point
soplex_pds : 331 simulation point
soplex_ref : 272 simulation point
povray : 370 simulation point
calculix : 398 simulation point
GemsFDTD : 244 simulation point
tonto : 336 simulation point
lbm : 1088 simulation point
wrf : 1166 simulation point
sphinx3 : 322 simulation point

bt.C.x의 simulation point는 그대로 사용하면 된다.

저녁 9시 30분에 checkpoint의 생성이 모두 끝남.

Advertisements
Posted in 1) Memo

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

누적 방문자 수
  • 88,610 hits
%d bloggers like this: