Paper Summarization_0.Abstract - PAIka2k/Sketch-to-Photo-Synthesis GitHub Wiki

Adversarial Open Domain Adaption for Sketch-to-Photo Synthesis


In this paper, we explore the open-domain sketch-to-photo translation, which aims to synthesize a realistic photo from a freehand sketch with its class label, even if the sketches of that class are missing in the training data.

์ด ๋…ผ๋ฌธ์—์„œ, ์šฐ๋ฆฌ๋Š” open-domain(training-set ์ด์™ธ์˜ data์—๋„ ์ ์šฉ์ด ๊ฐ€๋Šฅํ•œ) ์Šค์ผ€์น˜-์‚ฌ์ง„ ๋ณ€ํ™˜์— ๋Œ€ํ•ด ์‚ดํŽด๋ณผ ๊ฒƒ์ž…๋‹ˆ๋‹ค. ์Šค์ผ€์น˜-์‚ฌ์ง„ ๋ณ€ํ™˜์€ ํด๋ž˜์Šค Label์ด ๋ถ™์–ด์žˆ๋Š” ์†์œผ๋กœ ๊ทธ๋ฆฐ ์Šค์ผ€์น˜๋กœ ๋ถ€ํ„ฐ ์‹ค์ œ์™€ ๊ฐ™์€ ์‚ฌ์ง„์„ ํ•ฉ์„ฑํ•˜๋Š”๋ฐ์— ์ค‘์ ์„ ๋‘๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค.

It is challenging due to the lack of training supervision and the large geometry distortion between the freehand sketch and photo domains.

์Šค์ผ€์น˜-์‚ฌ์ง„ ๋ณ€ํ™˜(It)์€ training supervision์˜ ๋ถ€์กฑํ•จ, ์‚ฌ์ง„๊ณผ ์Šค์ผ€์น˜๊ฐ„์˜ ๊ธฐํ•˜ํ•™์  ์™œ๊ณก๋•Œ๋ฌธ์— ์–ด๋ ค์šด ๋ฌธ์ œ์ž…๋‹ˆ๋‹ค. -> supervision : ๊ฐ๋… // ํ•™์Šต์ง€๋„์˜ ๊ฐœ์„ ์„ ์œ„ํ•˜์—ฌ ์ œ๊ณต๋˜๋Š” ์ง€๋„ยท์กฐ์–ธ

To synthesize the absent freehand sketches from photos, we propose a framework that jointly learns sketch-to-photo and photo-to-sketch generation.

์‚ฌ์ง„์œผ๋กœ๋ถ€ํ„ฐ ์กด์žฌํ•˜์ง€ ์•Š๋Š” ์Šค์ผ€์น˜๋ฅผ ์ƒ์„ฑํ•˜๊ธฐ ์œ„ํ•ด, ์šฐ๋ฆฌ๋Š” ์Šค์ผ€์น˜๋กœ๋ถ€ํ„ฐ ์‚ฌ์ง„ ์ƒ์„ฑ๊ณผ ์‚ฌ์ง„์œผ๋กœ๋ถ€ํ„ฐ ์Šค์ผ€์น˜ ์ƒ์„ฑ์„ ๋™์‹œ์— ํ•™์Šตํ•˜๋Š” ํ”„๋ ˆ์ž„์›Œํฌ๋ฅผ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค.

However, the generator trained from fake sketches might lead to unsatisfying results when dealing with sketches of missing classes, due to the domain gap between synthesized sketches and real ones.

๊ทธ๋Ÿฌ๋‚˜, ๊ฐ€์งœ ์Šค์ผ€์น˜๋กœ๋ถ€ํ„ฐ ํ•™์Šต๋œ ์ƒ์„ฑ๊ธฐ๋Š” ์กด์žฌํ•˜์ง€ ์•Š๋Š” class์˜ ์Šค์ผ€์น˜๋ฅผ ๋‹ค๋ฃฐ ๋•Œ ๋งŒ์กฑ์Šค๋Ÿฝ์ง€ ๋ชปํ•œ ๊ฒฐ๊ณผ๋ฅผ ๋‚ณ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์™œ๋ƒํ•˜๋ฉด ์ƒ์„ฑ๋œ ์Šค์ผ€์น˜์™€ ์‹ค์ œ ์Šค์ผ€์น˜ ๊ฐ„์˜ domain gap์ด ์กด์žฌํ•˜๊ธฐ ๋•Œ๋ฌธ์ด์ฃ .

To alleviate this issue, we further propose a simple yet effective open-domain sampling and optimization strategy to โ€œfoolโ€ the generator into treating fake sketches as real ones.

์œ„์˜ ๋ฌธ์ œ(์‹ค์ œ ์Šค์ผ€์น˜์™€ ์ƒ์„ฑ๋œ ์Šค์ผ€์น˜ ๊ฐ„ domain gap)๋ฅผ ์™„ํ™”ํ•˜๊ธฐ ์œ„ํ•ด, ๋‹จ์ˆœํ•˜์ง€๋งŒ ํšจ๊ณผ์ ์ธ open-domain sampling๊ณผ ๊ฐ€์งœ ์Šค์ผ€์น˜๋ฅผ ์‹ค์ œ ์Šค์ผ€์น˜์ฒ˜๋Ÿผ ๊ฐ„์ฃผํ•˜๋„๋ก ์ƒ์„ฑ๊ธฐ๋ฅผ "๋ฉ์ฒญํ•˜๊ฒŒ" ์ตœ์ ํ™” ์‹œํ‚ค๋Š” ์ „๋žต์„ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค.

Our method takes advantage of the learned sketch-to-photo and photo-to-sketch mapping of in-domain data and generalizes them to the open-domain classes.

ํ•ด๋‹น ๋ชจ๋ธ์€ in-domain ๋ฐ์ดํ„ฐ๋“ค๋กœ ํ•™์Šต๋œ ์Šค์ผ€์น˜-๋„๋ฉด / ๋„๋ฉด-์Šค์ผ€์น˜ ๋งตํ•‘์„ open-domain classes๋กœ ์ผ๋ฐ˜ํ™”ํ•˜๋Š”๋ฐ์— ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

We validate our method on the Scribble and SketchyCOCO datasets.

ํ•ด๋‹น ๋ชจ๋ธ์„ "Scribble" ๋ฐ์ดํ„ฐ์…‹๊ณผ "SketchyCOCO" ๋ฐ์ดํ„ฐ์…‹์œผ๋กœ ๊ฒ€์ฆํ–ˆ์Šต๋‹ˆ๋‹ค.

Compared with the recent competing methods, our approach shows impressive results in synthesizing realistic color, texture, and maintaining the geometric composition for various categories of open-domain sketches.

์ตœ๊ทผ ๋น„๊ตํ• ๋งŒํ•œ ๋ชจ๋ธ๋“ค๊ณผ ๋น„๊ตํ–ˆ์„ ๋•Œ, ํ•ด๋‹น ๋ชจ๋ธ์€ ์‹ค์ œ ์ƒ‰๊ณผ ์งˆ๊ฐ์„ ์ƒ์„ฑํ•˜๊ณ  ๋‹ค์–‘ํ•œ ์นดํ…Œ๊ณ ๋ฆฌ์˜ open-domain ์Šค์ผ€์น˜๋“ค์˜ ๊ธฐํ•˜ํ•™์  ๊ตฌ์„ฑ์„ ์œ ์ง€ํ•˜๋Š”๋ฐ์— ์žˆ์–ด ์ข‹์€ ๊ฒฐ๊ณผ๋ฅผ ์–ป์—ˆ์Šต๋‹ˆ๋‹ค.